Question

在序列化具有大约250个属性和大约20,000行的动态对象时，我收到以下错误。当属性数量大约为20时，相同的代码工作正常。错误发生在 Serializer.Serialize（stream，lst）;

An unhandled exception of type 'System.OutOfMemoryException' occurred in System.ServiceModel.Internals.dll

at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at ProtoBuf.ProtoWriter.Flush(ProtoWriter writer) in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 534
at ProtoBuf.ProtoWriter.Dispose() in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 478
at ProtoBuf.ProtoWriter.System.IDisposable.Dispose() in c:\Dev\protobuf-net\protobuf-net\ProtoWriter.cs:line 472
at ProtoBuf.Meta.TypeModel.Serialize(Stream dest, Object value, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 218
at ProtoBuf.Meta.TypeModel.Serialize(Stream dest, Object value) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 201
at ProtoBuf.Serializer.Serialize[T](Stream destination, T instance) in c:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 87
at WcfService1.DynamicWrapper.Serialize(DynamicWrapper lst) in c:\Users\rkohli\Documents\Visual Studio 2012\Projects\WindowsFormsApplication3\WcfService1\SerializeObject.cs:line 136
at WcfService1.Service1.GetData(String sVisibleColumnList) in c:\Users\rkohli\Documents\Visual Studio 2012\Projects\WindowsFormsApplication3\WcfService1\Service1.svc.cs:line 22
at SyncInvokeGetData(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Dispatcher.ChannelHandler.OnContinueAsyncReceive(Object state)
at System.Runtime.ActionItem.DefaultActionItem.TraceAndInvoke()
at System.Runtime.ActionItem.DefaultActionItem.Invoke()
at System.Runtime.ActionItem.CallbackHelper.InvokeWithoutContext(Object state)
at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

以下是代码示例。

[Serializable]
[ProtoContract]
public class DynamicWrapper
{
    [ProtoMember(1, DataFormat = DataFormat.Group)]
    public List<DictWrapper> Items { get; set; }

    public DynamicWrapper()
    {
        Items = new List<DictWrapper>();
    }

    public static byte[] Serialize(DynamicWrapper lst)
    {
        byte[] msgOut;

        using (var stream = new MemoryStream())
        {
            Serializer.Serialize(stream, lst);
            msgOut = stream.ToArray();
        }

        return msgOut;
    }

    public static DynamicWrapper Deserialize(byte[] message)
    {
        DynamicWrapper msgOut;

        using (var stream = new MemoryStream(message))
        {
            msgOut = Serializer.Deserialize<DynamicWrapper>(stream);
        }

        return msgOut;
    }
}

[Serializable]
[ProtoContract]
public class DictWrapper
{
    [ProtoMember(1, DataFormat = DataFormat.Group)]
    public Dictionary<string, string > Dictionary { get; set; }

    public DictWrapper()
    {
        Dictionary = new Dictionary<string, string>();
    }
}

Answer 1

这里没有什么神奇之处。基于代码和单独发送的工作项目 - 数据简单：大。 120446305字节，准确（基于样本数据）。这里的主要问题是你正在使用字符串属性名称，并一遍又一遍地重复它们。现在，protobuf-net可以支持字符串缓存和重用，但默认情况下不会这样做 - 并且没有简单的方法将它应用于Dictionary<string,string>。但坦率地说，在我找出疯狂的方法使其在这种情况下起作用之前（这必然是一个突破性的变化）我必须首先指出这个根本不适合protobuf 。 Protobuf不提供“总是更小”的保证：它可以为典型的场景做好，即你的架构是预先知道的并且是可预测的。这个特定场景的所有内容都不是。

实际上，在给出的示例中，它正在加载来自DataSet的数据 - 原始数据中的数据仅为284MB。你在这里使用protobuf-net，对于不是针对的情况，导致了4倍的增长。

坦率地说，您最好发送原始DataSet有效负载。甚至更好：将数据集切换到二进制模式并发送，（162 MB）。

using (var file = File.Create("binary-ds")) { dataSet.WriteXml(file, XmlWriteMode.WriteSchema); }

或者更好 - 将其切换为二进制模式并通过gzip 运行，总计15MB：

using (var file = File.Create("binary-ds")) using (var gzip = new GZipStream(file, CompressionLevel.Optimal)) { dataSet.WriteXml(gzip, XmlWriteMode.WriteSchema); }

如果我使用常规POCO / DTO类重新编写示例并通过protobuf-net运行，我怀疑结果会相似（但没有DataTable的所有开销在这里），但是没有一种简单的方法可以改变你的场景，以便在不必改变数据布局的情况下很好地使用protobuf-net。如果您需要更改数据布局 - 上面的内容要容易得多。

ProtoBuf-Net错误消息 - “mscorlib.dll中发生类型'System.OutOfMemoryException'的异常，但未在用户代码中处理”

1 个答案: