Question

我正在使用OWIN自托管Web API 2 Windows服务。它在大多数情况下运行良好，除了在客户端（winforms app）上导致OutOfMemoryException的大型自定义对象。

问题：如何发布大型自定义对象？

OutOfMemoryException最初发生在JsonConvert.SerializeObject的这段代码的末尾：

using Newtonsoft.Json;

static HttpClient _httpClient = new HttpClient();  

public async Task SaveMyObjectAsync(MyObject largeObject)
{
    var response = await _httpClient.PostAsync("myobjects/route/", new JsonContent(largeObject));
    response.EnsureSuccessStatusCode();
}

public static class JsonSettings
{
    public static readonly JsonSerializerSettings Default =
    new JsonSerializerSettings
    {
        ContractResolver = new DefaultContractResolver(),
        NullValueHandling = NullValueHandling.Ignore,
        ReferenceLoopHandling = ReferenceLoopHandling.Serialize,
        DateTimeZoneHandling = DateTimeZoneHandling.RoundtripKind,
        DateParseHandling = DateParseHandling.DateTimeOffset,
        DateFormatHandling = DateFormatHandling.IsoDateFormat,
        Formatting = Formatting.Indented,
        Converters = new List<JsonConverter>
        {
            new StringEnumConverter(),
        }
    };
}

public class JsonContent : StringContent
{
    public JsonContent(object value) : base(JsonConvert.SerializeObject(value, JsonSettings.Default), Encoding.UTF8, "application/json") {}
}

第一次尝试

所以从this answer我换掉了serialize方法来代替写入本地文件。这工作了一段时间，直到我意识到它刚刚增加了它可以处理的大小限制。我仍然得到一个包含更大对象的OutOfMemoryException但现在它在File.ReadAllText上

public JsonContent(object value) : base(SerializeObjectByStream(value), Encoding.UTF8, "application/json") { }

static string SerializeObjectByStream(object value)
{
    using (TextWriter textWriter = File.CreateText("LocalJsonFile.json"))
    {
        JsonConvert.DefaultSettings = () => JsonSettings.Default;
        using (var jsonWriter = new JsonTextWriter(textWriter))
        {
            var serializer = new JsonSerializer();
            serializer.Serialize(jsonWriter, value);
            jsonWriter.Flush();
        }
    }
    return File.ReadAllText("LocalJsonFile.json");
}

多部分尝试

无论如何，将这么大的对象发送到一个部分可能是一个坏主意，所以我尝试使用MultipartContent来分解它。大多数示例似乎涵盖了阅读多部分请求，而不是创建它，但此代码适用于我的常规大小自定义对象。不幸的是，STILL会为大对象抛出OutOfMemoryException。这次它是ser.Serialize(jsonWriter, request)的Newtonsoft JsonSerializer的内部。

我也尝试过使用FileStream而不是MemoryStream来解决同样的问题。这次OutOfMemoryException位于_httpClient.PostAsync

using (var content = new MultipartContent())
{
    using (var stream = new MemoryStream())
    {
        var writer = new StreamWriter(stream);
        JsonConvert.DefaultSettings = () => JsonSettings.Default;
        var jsonWriter = new JsonTextWriter(writer);
        var ser = new JsonSerializer();
        ser.Serialize(jsonWriter, request);
        jsonWriter.Flush();
        stream.Seek(0, SeekOrigin.Begin);
        content.Add(new StreamContent(stream));

        var response = await _httpClient.PostAsync("myobjects/route/", content);
        response.EnsureSuccessStatusCode();
    }
}

我正在做的就是推动这些数据在不同的地方找到内存不足的问题如何将这个大型自定义对象分解为块 - 同时将其保存在1个事务中????

Answer 1

那比我想象的要简单得多。这段代码似乎表现得相当好（为了简单起见，取消了取消等等）。虽然我有点担心我的FileStream使用通用文件名。异步我认为有可能2个实例可能会同时尝试创建同一个文件吗？

从JsonSerializer切换到BinaryFormatter非常痛苦，因为这意味着我必须通过[Serializable]属性来完成并装饰我的所有类。我也依赖于JsonSerializer的行为来使用默认的构造函数as described here，它将我的空值转换为空字符串。

<强>客户端：

const int MaximumChunkSize = 1024000;

public async Task SaveMyObjectAsync(MyObject largeObject)
{
    //use fileStream to write the object to disk, 
    // so it does not hold it all in memory at the same time
    using (var stream = new FileStream("LocalStreamFile.json", FileMode.Create))
    {
        //JsonSerializer unable to serialize a large object 
        // without OOM error, so use BinaryFormatter
        var formatter = new BinaryFormatter();
        formatter.Serialize(stream, request);
        //return to the start of the stream
        stream.Seek(0, SeekOrigin.Begin);
        using (var content = new MultipartContent())
        {
            var buffer = new byte[MaximumChunkSize];
            while (stream.Read(buffer, 0, buffer.Length) > 0)
            {
                //add the large object in chunks to the multipart content
                content.Add(new JsonContent(buffer));
            }
            var response = await _httpClient.PostAsync("myobjects/route/", content);
            response.EnsureSuccessStatusCode();
        }
    }
}

服务器

[HttpPost, Route("myobjects/route/")] public async Task<IHttpActionResult> SaveMyObjectAsync() { if (Request.Content.IsMimeMultipartContent() == false) { return StatusCode(HttpStatusCode.BadRequest); } var contentStreamProvider = await Request.Content.ReadAsMultipartAsync(); var stream = new FileStream("LocalStreamFile.json", FileMode.Create); foreach (var content in contentStreamProvider.Contents) { //read out the chunk and convert from json var requestArray = JsonConvert.DeserializeObject<byte[]>(await content.ReadAsStringAsync(), JsonSettings.Default); stream.Write(requestArray, 0, requestArray.Length); } stream.Seek(0, SeekOrigin.Begin); var formatter = new BinaryFormatter(); //convert back from stream to our original large object var request = (MyObject)formatter.Deserialize(stream); //save to database etc ... return Ok(); }

使用大对象的Owin WebApi Post方法

第一次尝试

多部分尝试

1 个答案: