Question

我的代码中有ConcurrentBag<Point3DCollection>。

我正在试图弄清楚如何序列化它们。当然，我可以使用提供者模型类进行迭代或打包，但我不知道它是否已经完成。

Point3DCollection本身可能非常大，可以进行压缩以加快磁盘的读写速度，但我需要的响应时间主要是用户界面规模。换句话说，出于性能原因，我更喜欢二进制格式而不是XAML文本格式。（有一个很好的XAML文本序列化器，它是Helix 3D CodeProject的一部分，但它比我想要的慢。）

这是一个用例，我还在推出自己的序列化程序，还是那些已经打包过这类数据的东西？

Answer 1

以下是一些处理Point3DCollection行包的字符串和二进制序列化的扩展方法。正如我在评论中所说，我不认为在所有情况下都有最好的方法，所以你可能想尝试两种方法。另请注意，他们使用Stream参数作为输入，因此您可以通过调用GZipStream的DeflateStream链接这些参数。

public static class Point3DExtensions
{
    public static void StringSerialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        StreamWriter writer = new StreamWriter(stream);
        Point3DCollectionConverter converter = new Point3DCollectionConverter();
        foreach (Point3DCollection coll in bag)
        {
            // we need to use the english locale as the converter needs that for parsing...
            string line = (string)converter.ConvertTo(null, CultureInfo.GetCultureInfo("en-US"), coll, typeof(string));
            writer.WriteLine(line);
        }
        writer.Flush();
    }

    public static void StringDeserialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        StreamReader reader = new StreamReader(stream);
        Point3DCollectionConverter converter = new Point3DCollectionConverter();
        do
        {
            string line = reader.ReadLine();
            if (line == null)
                break;

            bag.Add((Point3DCollection)converter.ConvertFrom(line));

            // NOTE: could also use this:
            //bag.Add(Point3DCollection.Parse(line));
        }
        while (true);
    }

    public static void BinarySerialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        BinaryWriter writer = new BinaryWriter(stream);
        writer.Write(bag.Count);
        foreach (Point3DCollection coll in bag)
        {
            writer.Write(coll.Count);
            foreach (Point3D point in coll)
            {
                writer.Write(point.X);
                writer.Write(point.Y);
                writer.Write(point.Z);
            }
        }
        writer.Flush();
    }

    public static void BinaryDeserialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        BinaryReader reader = new BinaryReader(stream);
        int count = reader.ReadInt32();
        for (int i = 0; i < count; i++)
        {
            int pointCount = reader.ReadInt32();
            Point3DCollection coll = new Point3DCollection(pointCount);
            for (int j = 0; j < pointCount; j++)
            {
                coll.Add(new Point3D(reader.ReadDouble(), reader.ReadDouble(), reader.ReadDouble()));
            }
            bag.Add(coll);
        }
    }
}

还有一个小的控制台应用程序测试程序：

    static void Main(string[] args)
    {
        Random rand = new Random(Environment.TickCount);
        ConcurrentBag<Point3DCollection> bag = new ConcurrentBag<Point3DCollection>();
        for (int i = 0; i < 100; i++)
        {
            Point3DCollection coll = new Point3DCollection();
            bag.Add(coll);

            for (int j = rand.Next(10); j < rand.Next(100); j++)
            {
                Point3D point = new Point3D(rand.NextDouble(), rand.NextDouble(), rand.NextDouble());
                coll.Add(point);
            }
        }

        using (FileStream stream = new FileStream("test.bin", FileMode.Create))
        {
            bag.StringSerialize(stream); // or Binary
        }

        ConcurrentBag<Point3DCollection> newbag = new ConcurrentBag<Point3DCollection>();
        using (FileStream stream = new FileStream("test.bin", FileMode.Open))
        {
            newbag.StringDeserialize(stream); // or Binary
            foreach (Point3DCollection coll in newbag)
            {
                foreach (Point3D point in coll)
                {
                    Console.WriteLine(point);
                }
                Console.WriteLine();
            }
        }
    }
}

Answer 2

压缩可能会利用重复的坐标。序列化程序通常也会使用重复对象的引用，虽然我不确定有多少设置可以使用结构（如Point3D）。无论如何，这里有一些如何序列化这个的例子。要使用标准格式化程序，您需要将数据类型转换为大多数支持的类型：list / array。下面的代码使用Nuget包NUnit和Json.NET。

using Newtonsoft.Json;
using Newtonsoft.Json.Bson;
using NUnit.Framework;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Windows.Media.Media3D;

namespace DemoPoint3DSerialize
{
    [TestFixture]
    class Tests
    {
        [Test]
        public void DemoBinary()
        {
            // this shows how to convert them all to strings
            var collection = CreateCollection();
            var data = collection.Select(c => c.ToArray()).ToList(); // switch to serializable types
            var formatter = new BinaryFormatter();

            using (var ms = new MemoryStream())
            {
                formatter.Serialize(ms, data);
                Trace.WriteLine("Binary of Array Size: " + ms.Position);
                ms.Position = 0;
                var dupe = (List<Point3D[]>)formatter.Deserialize(ms);
                var result = new ConcurrentBag<Point3DCollection>(dupe.Select(r => new Point3DCollection(r)));
                VerifyEquality(collection, result);
            }
        }

        [Test]
        public void DemoString()
        {
            // this shows how to convert them all to strings
            var collection = CreateCollection();
            IEnumerable<IList<Point3D>> tmp = collection;
            var strings = collection.Select(c => c.ToString()).ToList();

            Trace.WriteLine("String Size: " + strings.Sum(s => s.Length)); // eh, 2x for Unicode
            var result = new ConcurrentBag<Point3DCollection>(strings.Select(r => Point3DCollection.Parse(r)));

            VerifyEquality(collection, result);
        }

        [Test]
        public void DemoDeflateString()
        {
            // this shows how to convert them all to strings
            var collection = CreateCollection();
            var formatter = new BinaryFormatter(); // not really helping much: could 
            var strings = collection.Select(c => c.ToString()).ToList();

            using (var ms = new MemoryStream())
            {
                using (var def = new DeflateStream(ms, CompressionLevel.Optimal, true))
                {
                    formatter.Serialize(def, strings);
                }
                Trace.WriteLine("Deflate Size: " + ms.Position);
                ms.Position = 0;
                using (var def = new DeflateStream(ms, CompressionMode.Decompress))
                {
                    var stringsDupe = (IList<string>)formatter.Deserialize(def);
                    var result = new ConcurrentBag<Point3DCollection>(stringsDupe.Select(r => Point3DCollection.Parse(r)));

                    VerifyEquality(collection, result);
                }
            }
        }

        [Test]
        public void DemoStraightJson()
        {
            // this uses Json.NET
            var collection = CreateCollection();
            var formatter = new JsonSerializer();

            using (var ms = new MemoryStream())
            {
                using (var stream = new StreamWriter(ms, new UTF8Encoding(true), 2048, true))
                using (var writer = new JsonTextWriter(stream))
                {
                    formatter.Serialize(writer, collection);
                }
                Trace.WriteLine("JSON Size: " + ms.Position);
                ms.Position = 0;
                using (var stream = new StreamReader(ms))
                using (var reader = new JsonTextReader(stream))
                {
                    var result = formatter.Deserialize<List<Point3DCollection>>(reader);
                    VerifyEquality(collection, new ConcurrentBag<Point3DCollection>(result));
                }
            }
        }

        [Test]
        public void DemoBsonOfArray()
        {
            // this uses Json.NET
            var collection = CreateCollection();
            var formatter = new JsonSerializer();

            using (var ms = new MemoryStream())
            {
                using (var stream = new BinaryWriter(ms, new UTF8Encoding(true), true))
                using (var writer = new BsonWriter(stream))
                {
                    formatter.Serialize(writer, collection);
                }
                Trace.WriteLine("BSON Size: " + ms.Position);
                ms.Position = 0;
                using (var stream = new BinaryReader(ms))
                using (var reader = new BsonReader(stream, true, DateTimeKind.Unspecified))
                {
                    var result = formatter.Deserialize<List<Point3DCollection>>(reader); // doesn't seem to read out that concurrentBag
                    VerifyEquality(collection, new ConcurrentBag<Point3DCollection>(result));
                }
            }
        }

        private ConcurrentBag<Point3DCollection> CreateCollection()
        {
            var rand = new Random(42);
            var bag = new ConcurrentBag<Point3DCollection>();

            for (int i = 0; i < 10; i++)
            {
                var collection = new Point3DCollection();
                for (int j = 0; j < i + 10; j++)
                {
                    var point = new Point3D(rand.NextDouble(), rand.NextDouble(), rand.NextDouble());
                    collection.Add(point);
                }
                bag.Add(collection);
            }
            return bag;
        }

        private class CollectionComparer : IEqualityComparer<Point3DCollection>
        {
            public bool Equals(Point3DCollection x, Point3DCollection y)
            {
                return x.SequenceEqual(y);
            }

            public int GetHashCode(Point3DCollection obj)
            {
                return obj.GetHashCode();
            }
        }

        private void VerifyEquality(ConcurrentBag<Point3DCollection> collection, ConcurrentBag<Point3DCollection> result)
        {
            var first = collection.OrderBy(c => c.Count);
            var second = collection.OrderBy(c => c.Count);
            first.SequenceEqual(second, new CollectionComparer());
        }


    }
}

Answer 3

使用Google的protobuf-net。 protobuf-net是Google协议缓冲区二进制序列化格式的开源.net实现，可用作BinaryFormatter序列化程序的替代品。它可能是最快的解决方案，也是最容易实现的。

这是一个指向protobuf-net的主要谷歌wiki的链接。在左侧，您可以找到所有最新二进制文件的下载。

https://code.google.com/p/protobuf-net/

这是一篇很棒的文章，您可能希望先了解它，以了解它的工作原理。

http://wallaceturner.com/serialization-with-protobuf-net

以下链接指向Google维基上有关您的具体问题的讨论。答案在页面底部。这就是我获得下面的代码并用你帖子中的详细信息替换的地方。

https://code.google.com/p/protobuf-net/issues/detail?id=354

我自己没有使用它，但它看起来是一个非常好的解决方案，满足您的需求。根据我收集的内容，您的代码最终会有一些变化。

[ProtoContract]
public class MyClass {
    public ConcurrentQueue<Point3DCollection> Points {get;set;}

    [ProtoMember(1)]
    private Point3DCollection[] Items
    {
        get { return Points.ToArray(); }
        set { Items = new ConcurrentBag<Point3DCollection>(value); }
    }
}

祝你好运。小心。

Answer 4

对于大量数据，为什么不考虑Sqlite或任何其他小型数据库系统等，它们可以在文件中存储结构化数据。

我见过许多使用数据库的3d程序来存储结构以及关系，这使得它们可以部分地插入/更新/删除数据。

Sqlite / database的好处是多线程序列化以提高速度，但是你需要在sqlite上做一点点工作来启用多线程sqlite连接，否则你可以使用SQL Express的LocalDB甚至是Sql Compact。

还可以通过查询完成一些加载数据的工作量，这些查询将很好地被数据库索引。大多数事情都可以在后台工作人员上完成，而不会干扰用户界面。

Sqlite具有有限的多线程支持，可在此处探讨http://www.sqlite.org/threadsafe.html

Sql Compact是线程安全的，需要安装，无需管理员权限即可安装。您也可以使用Entity框架。

序列化XAML的ConcurrentBag

4 个答案: