Question

我正在尝试设置一个“外部” Web api，该API可以接收较大的http帖子（+ 1GB），并将流转发到另一个将请求内容写入文件的“内部” web api。我基于使用自定义WebHostBufferPolicySelector以及在控制器方法中使用UseBufferedInputStream方法的示例对实现进行了建模。使用IIS Express时，它可以按预期工作，内存占用没有明显增加，但是，只要将我的代码部署到IIS，内存占用就很大，并导致OOM。

我已将跟踪语句放入控制器方法和WebHostBufferPolicySelector.UseBufferedInputStream中，并已验证UseBufferedInputStream始终返回false且控制器方法受到攻击。我注意到的唯一区别是，当我调试时，UseBufferedInputStream和我的控制器方法之间的时间戳非常接近。在IIS上托管的时间戳之间相距甚远，这表明调用UseBufferedInputStream和调用我的控制器方法之间的某些时间完全缓冲了请求。

我正在寻找一些提示，以找出导致请求被缓冲的原因，以及如何不对请求进行缓冲并一直使用流式传输。

客户端进入外部Web api，其内容类型为应用程序/八位字节流，并且具有“已阻止的传输编码”。

用于构建实施

https://forums.asp.net/t/2018289.aspx?Web+API2+WebHostBufferPolicySelector+UseBufferedInputStream+override

https://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/

代理Web Api控制器方法

        <HttpPost, Route("postLargeFile")>
        Protected Overridable Async Function PostLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
            Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Started {0}", NameOf(MyProxyController.PostLargeFile))

            Dim internalHttpClient As HttpClient
            Dim fowardingContent As StreamContent = Nothing
            Dim fowardingMessage As HttpRequestMessage = Nothing
            Dim fowardingResponse As HttpResponseMessage = Nothing
            Dim externalResponse As HttpResponseMessage = Nothing

            Try
                internalHttpClient = New HttpClient()
                internalHttpClient.BaseAddress = "https://myinternalService.com"

                fowardingMessage = New HttpRequestMessage(HttpMethod.Post, "https://myinternalService.com/saveLargeFile")
                fowardingContent = New StreamContent(HttpContext.Current.Request.GetBufferlessInputStream(True))
                CopyContentHeaders(Request.Content, fowardingContent)

                fowardingMessage.Headers.TransferEncodingChunked = True
                fowardingMessage.Content = fowardingContent

                fowardingResponse = Await internalHttpClient.SendAsync(fowardingMessage, HttpCompletionOption.ResponseHeadersRead)

                externalResponse = New HttpResponseMessage(fowardingResponse.StatusCode)
                externalResponse.Content = New StreamContent(Await fowardingResponse.Content.ReadAsStreamAsync)
                CopyContentHeaders(fowardingResponse.Content, externalResponse.Content)

                Return New Results.ResponseMessageResult(externalResponse)

            Catch ex As Exception
                Return InternalServerError(ex)
            Finally
                Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Finished {0}", NameOf(MyProxyController.PostLargeFile))
            End Try
        End Function

内部Web Api控制器方法

        <HttpPost, Route("saveLargeFile")>
        Protected Overridable Async Function SaveLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
            Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Started {0}", NameOf(MyInternalController.PostLargeFile))

            Dim bufferlessStream As IO.Stream
            Dim fowardingContent As StreamContent = Nothing

            Try

                bufferlessStream = HttpContext.Current.Request.GetBufferlessInputStream()
                Using fileStream As IO.FileStream = IO.File.Create("MyFile.txt")
                    bufferlessStream.CopyTo(fileStream)
                    fileStream.Flush()
                End Using

                Return New Results.StatusCodeResult(Net.HttpStatusCode.Created, Me)

            Catch ex As Exception
                Return InternalServerError(ex)
            Finally
                Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Finished {0}", NameOf(MyInternalController.PostLargeFile))
            End Try
        End Function

策略选择器配置

Public Class MyBufferPolicySelector
    Inherits Http.WebHost.WebHostBufferPolicySelector

    Public Property Tracer As ITraceWriter

    Public Overrides Function UseBufferedInputStream(hostContext As Object) As Boolean
        UseBufferedInputStream = False
        Tracer?.Info(Nothing, $"{Me.GetType.Namespace}.{NameOf(MyBufferPolicySelector)}", "{0} UseBufferedInputStream={1}", HttpContext.Current?.Request?.Url?.AbsoluteUri, UseBufferedInputStream)
        Return UseBufferedInputStream
    End Function
End Class

用于内部和外部Web API的WebApiConfig

Public Module WebApiConfig

    Public Sub Register(ByVal config As HttpConfiguration)
        Dim tracer As SystemDiagnosticsTraceWriter

        ' Web API configuration and services

        ' Web API routes
        config.MapHttpAttributeRoutes()

        tracer = config.EnableSystemDiagnosticsTracing
        tracer.IsVerbose = True
        tracer.MinimumLevel = Tracing.TraceLevel.Debug

        GlobalConfiguration.Configuration.Services.Replace(GetType(IHostBufferPolicySelector), New MyBufferPolicySelector() With {.Tracer = tracer})

    End Sub

End Module

Answer 1

我能够弄清楚是什么导致了IIS中的缓冲。下面的链接将我引到IIS中的uploadReadAheadSize设置。这被最大化了。因此，这将导致IIS在将请求传递到存在Web API管道的模块（Web API控制器）之前完全读入/缓冲请求。将其设置为默认值之后，我发现我的大文件帖子没有得到缓冲，应用程序池的内存占用仍然很低，没有更多的内存不足异常，并且性能得到了大幅提升。太好了！

但是现在我遇到了以下链接中所述的相同问题。当需要SSL（在我们的非开发环境中需要在IIS中设置）时，需要增加uploadReadAheadSize，以便ssl可以在ssl模块中工作。它可能与某些SSL重新协商有关。

有人能描述一种方法来防止SSL缓冲以保持较低的内存占用并防止大型http帖子出现内存不足异常吗？

Large file upload when using ssl and client certificates (uploadReadAheadSize) but dont want all data to be readahead

控制器执行操作之前，IIS缓冲请求内容

1 个答案: