在keepAlive模式下了解HttpWebRequest

时间:2018-03-29 10:52:16

标签: vb.net httpwebrequest

我知道这个话题已经多次讨论过了 但我需要了解如何以正确的方式编写代码。

我使用协议版本HTTP 1.1的相同HttpWebRequest(到同一个url)使用了更多次。

Method =“POST”

KeepAlive = True

但每次我需要发送不同的请求,并得到不同的回复。

(注意。下一个代码,它不正确,并抛出异常)

Private Sub SendHttpWebReq()
    Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)
    httpWebReq.Method = "POST"
    httpWebReq.KeepAlive = True
    httpWebReq.ContentType = "application/x-www-form-urlencoded"
    Dim myRequestString As New List(Of String) From {"abc", "def"}
    Dim ContentList As New List(Of String)
    For a = 0 To 1
        Dim inputData As String = MyRequestString(a)
        Dim postData As String = "firstone" + ChrW(61) + inputData
        Dim encoding As New System.Text.ASCIIEncoding()
        Dim byteData As Byte() = encoding.GetBytes(postData)
        httpWebReq.ContentLength = byteData.Length
        Dim newStream As IO.Stream = httpWebReq.GetRequestStream()
        newStream.Write(byteData, 0, byteData.Length)
        newStream.Flush()
        newStream.Dispose()
        Dim Response As Net.WebResponse = httpWebReq.GetResponse()
        Dim ResponseStream As Io.Stream = Response.GetResponseStream()
        Dim Content = New Io.MemoryStream()
        ResponseStream.CopyTo(Content)
        Response.Close()
        Response.Dispose()
        ResponseStream.Flush()
        ResponseStream.Dispose()
        ContentList.Add(System.Text.Encoding.UTF8.GetString(Content.ToArray))
        Content = Nothing
    Next
End Sub

当我运行代码时,第一次得到正确的响应,但是当我尝试“重用”HttpWebRequest时,抛出异常

在这一行:

httpWebReq.ContentLength = byteData.Length

例外情况'写入开始后无法设置此属性'

搜索我发现了这个话题: Am I able to reuse a HttpWebRequest?

在解释为“重用”HttpWebRequest的情况下,必须关闭流和WebResponse,然后我发布它,释放资源。

同样在本主题中,它解释了同样的事情:https://social.msdn.microsoft.com/Forums/en-US/8efad6b3-bc75-48f0-9858-8115dabb85c8/reusing-httpwebrequest-object?forum=netfxnetcom

但在另一个主题中:This property cannot be set after writing has started! on a C# WebRequest Object

一名成员表示无法重复使用HttpWebRequest。

我在'重用'和'创建新'之间感到困惑。

我需要了解它所引用的'KeepAlive':连接或请求?

我想当我执行这条指令时:

Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)

我应该创建一个HttpWebRequest类的实例,

但是我应该与这条指令建立联系

Dim newStream As IO.Stream = httpWebReq.GetRequestStream()

我是对的吗?

任何帮助?

1 个答案:

答案 0 :(得分:1)

我认为这需要一些澄清,因为这种说法可能会被误导,它的表达方式如下:

  

1 WebRequest => 1 WebResponse。一旦初始化,您就无法在WebRequest中更改任何内容。

原则上这仍然有效,但初始化的术语可能会令人困惑。一个更好的定义是:

  

在发出请求并返回WebResponse之后,您无法更改WebRequest的任何参数,直到WebResponse关闭(处置)之后。

WebResponse返回结果后,可以显式或隐式地(在Using块中)关闭它 - 您可以请求另一个,根据需要修改WebRequest个参数(例如,将方法从POST更改为GET) 更多,在请求新的WebResponse 之前,必须重新初始化WebRequest。如果你不这样做,它就会回到它的默认值。

我在下面发布的代码是一个经典上下文(Web LogIn请求)的示例,当WebRequest必须在同一过程中多次重新初始化时,接收不确定数量的WebResponses,直到到达目的地地址(或着陆页)。

这或多或少是架构:

                  --------------
(GET or POST)     | WebRequest |       (Method is POST)
      |---------> | GET/(POST) | <-----------| <-------------- |
      |           --------------             |                 |
      |                 |                    |                 |
--------------    ---------------    ------------------   --------------
|    New     |    | WebResponse |--> | LogIn Required |-->|   LogIn    |
|  Location  |    ---------------    ------------------   |   Address  |
|   (Set     |          |                                 --------------
|  Referer)  |          |
--------------     (Set Cookies)
      |                 |
      |           ---------------
      |           |    LogIn    |
 Redirection <----|     OK      |---NO---|
                  ---------------        |
                        |                |
                       YES               |
                   (Set Cookies)         |
                        |             Request
                  ---------------     Denied
                  |  Response   |        |
                  |    URI      |        |
                  ---------------        |
                        |                |
                       EXIT <------------|
                        |

请注意,发出WebRequest,如果访问请求的资源URI需要身份验证,则服务器可能无法使用StatusCode 302(Found),301(Moved)或303(Redirected)回答,它可能只是将StatusCode设置为200(OK)。重定向是隐含的,因为&#34;位置&#34;标题已设置,或者如果它是WebForm登录,则检索到的Html页面包含重定向。

无论如何,在检测到重定向后,必须将新的重定向位置跟踪到目的地。重定向可能包含一个或多个Hops,通常必须手动解决(以确认我们已经发送到我们实际想要去的地方)。

<小时/> 关于keep-alive标题 客户端和/或服务器设置keep-alive标头,提示建立的连接应保持打开的对应方,至少在一段时间内,因为与当前交易相关的其他资源将被交换的机会很高。
这可以防止可能产生大量昂贵的连接。
此设置在HttpFtp请求中很常见,它是Http 1.1中的标准。

Hypertext Transfer Protocol (HTTP) Keep-Alive Header (IETF)
Compatibility with HTTP/1.0 Persistent Connections (IETF)

需要记住的是,keep-alive标题是指使用WebRequest建立的连接,而不是WebRequest本身。创建连接时指定应保持打开状态,后续请求应保持connection: keep-alive标头符合协议。
在.NET HttpWebRequest中,将KeepAlive属性设置为False,相当于设置connection: close标题。

但是,管理连接和允许进程访问的连接池的逻辑由ServicePointManager管理,使用ServicePoint作为每个连接请求的引用。< BR />

因此,WebRequest可以指定它要求创建的Connection应该保持打开(因为它需要多次重用),但是Connection背后的真正逻辑,如何建立,维护并管理,在其他地方撒谎。

在Http 2.0(StackOverflow / StackExchange使用此协议)中,出于这个原因,完全忽略了keep-alive设置。连接逻辑在更高,独立的级别进行管理。

  

(...)当你每3秒调用1个新的HttpWebRequest时,每10个   秒,还是每60秒?我发送这些内容的区别是什么   请求是真还是假?

您将KeepAlive属性设置为提示 Connection建立的Connection Manager应保持打开状态,因为您知道它将被重用。但是,如果正在使用的协议提供此设置并且在管理连接池复合体的内部逻辑所施加的限制内,ServicePointManager和远程服务器都将符合请求。
它应该在Http 1.0中设置,它是Http 1.1中的默认设置,在Http 2.0中被忽略。
由于您无法知道在建立连接之前将使用哪个协议,因此通常将其设置为keep-alive,因为到达请求的资源的路由中的某些设备(特定是代理)可能会需要此设置是明确的(阅读有关代理及其行为的IETF文档)。

<小时/> 在此示例中,WebRequest在循环中重复初始化,基础WebResponse每次都为Disposed,直到收到StatusCode 200(OK)或请求为止被拒绝或我们被重定向太多次(取消令牌在这里也可能有用)。

在示例中,主方法应以这种方式调用:

Public Async Sub SomeMethodAsync()

    LoginParameters = New LoginObject() With {
        .CookieJar = New CookieContainer,
        .LogInUrl = "[Some IP Address]",
        .Credentials = New Dictionary(Of String, String)
    }
    LoginParameters.Credentials.Add("UserName", "[Username]")
    LoginParameters.Credentials.Add("Email", "[email]")
    LoginParameters.Credentials.Add("Password", "[Password]")

    LoginParameters = Await HttpLogIn(LoginParameters)

End Sub

必须保留LoginParameters对象,因为引用了CookieContainer,其中包含身份验证后收到的Cookie。当初始化新的WebRequest时,这些Cookie将被传递到服务器,作为&#34;证明&#34;请求的凭据已经过身份验证。请注意,这些Cookie会在一段时间后过期(#34;刷新&#34;发布新的WebRequest时,除非会话有时间限制)。如果是这种情况,将自动重复登录过程。

Imports System.Net
Imports System.Net.Security
Imports System.IO
Imports System.Security
Imports System.Security.Cryptography
Imports System.Security.Cryptography.X509Certificates
Imports System.Text


Public LoginParameters As LoginObject

Public Class LoginObject
    Public Property LogInUrl As String
    Public Property ResponseUrl As String
    Public Property Credentials As Dictionary(Of String, String)
    Public Property StatusCode As HttpStatusCode
    Public Property CookieJar As New CookieContainer()
End Class


Public Async Function HttpLogIn(LogInParameters As LoginObject) As Task(Of LoginObject)

    Dim httpRequest As HttpWebRequest
    Dim StatusCode As HttpStatusCode
    Dim MaxHops As Integer = 20

    ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 Or
                                           SecurityProtocolType.Tls Or
                                           SecurityProtocolType.Tls11 Or
                                           SecurityProtocolType.Tls12
    ServicePointManager.ServerCertificateValidationCallback = AddressOf CertificateValidation

    httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)

    Try
        HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)
        Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
            StatusCode = httpResponse.StatusCode
        End Using

        If StatusCode = HttpStatusCode.OK OrElse StatusCode = HttpStatusCode.NoContent Then
            'POST Parameters are URLEncoded and the encoded strings converted to a Byte array of UTF8 chars
            Dim EncodedParameters As Byte() = HTTP_EncodePOSTParameters(LogInParameters.Credentials)

            httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)
            httpRequest.Method = WebRequestMethods.Http.Post
            httpRequest.ContentType = "application/x-www-form-urlencoded"
            httpRequest.ContentLength = EncodedParameters.Length
            HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)

            Using stream As Stream = Await httpRequest.GetRequestStreamAsync()
                stream.Write(EncodedParameters, 0, EncodedParameters.Length)
            End Using


            Dim Hops As Integer = 0
            Dim Referer As String = LogInParameters.LogInUrl
            Dim LastHttpMethod As String = httpRequest.Method

            Do
                'Evaluate Authentication redirect or page moved
                Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
                    StatusCode = httpResponse.StatusCode
                    LogInParameters.ResponseUrl = URIFromResponseLocation(httpResponse).ToString()
                End Using

                If (StatusCode = HttpStatusCode.Moved) OrElse
                   (StatusCode = HttpStatusCode.Found) OrElse
                   (StatusCode = HttpStatusCode.RedirectMethod) OrElse
                   (StatusCode = HttpStatusCode.RedirectKeepVerb) Then

                    httpRequest = WebRequest.CreateHttp(LogInParameters.ResponseUrl)
                    HTTP_RequestHeadersInit(httpRequest, Referer, LogInParameters.CookieJar)
                    If StatusCode = HttpStatusCode.RedirectKeepVerb Then
                        httpRequest.Method = LastHttpMethod
                    Else
                        LastHttpMethod = httpRequest.Method
                    End If
                End If

                If (CType(StatusCode, Integer) > 320) OrElse Hops >= MaxHops Then
                    Exit Do
                End If
                Hops += 1
            Loop While (StatusCode <> HttpStatusCode.OK)

            If StatusCode = HttpStatusCode.OK Then
                LogInParameters.CookieJar = httpRequest.CookieContainer
            End If

        End If

    Catch exW As WebException
        StatusCode = If(exW.Response IsNot Nothing,
                        CType(exW.Response, HttpWebResponse).StatusCode,
                        CType(exW.Status, HttpStatusCode))

    Catch exS As System.Exception
        StatusCode = CType(WebExceptionStatus.RequestCanceled, HttpStatusCode)

    Finally
        ServicePointManager.ServerCertificateValidationCallback = Nothing
    End Try

    LogInParameters.StatusCode = StatusCode
    Return LogInParameters

End Function

Private Sub HTTP_RequestHeadersInit(ByRef httpReq As HttpWebRequest,
                                            Referer As String,
                                            CookiesJar As CookieContainer)

    httpReq.Date = DateTime.Now
    httpReq.CookieContainer = CookiesJar
    httpReq.KeepAlive = True
    httpReq.ConnectionGroupName = Guid.NewGuid().ToString()
    httpReq.AllowAutoRedirect = False
    httpReq.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
    httpReq.ServicePoint.Expect100Continue = False
    httpReq.Referer = Referer
    httpReq.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"
    httpReq.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    httpReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.9,en;q=0.5")
    httpReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
    httpReq.Headers.Add(HttpRequestHeader.CacheControl, "no-cache")
End Sub

Private Function HTTP_EncodePOSTParameters(PostParameters As Dictionary(Of String, String)) As Byte()

    Dim Encoder As New System.Text.UTF8Encoding()
    Dim CredentialValues As New StringBuilder()
    Dim _first As Boolean = True

    For Each CurrentKeyPair As KeyValuePair(Of String, String) In PostParameters
        If _first = False Then CredentialValues.Append("&")
        CredentialValues.AppendFormat("{0}={1}", WebUtility.UrlEncode(CurrentKeyPair.Key),
                                                 WebUtility.UrlEncode(CurrentKeyPair.Value))
        _first = False
    Next

    Return Encoder.GetBytes(CredentialValues.ToString())

End Function

Private Function URIFromResponseLocation(Response As HttpWebResponse) As System.Uri
    Dim uri As Uri
    Dim Location As String = Response.Headers("Location")

    Try
        If uri.IsWellFormedUriString(Location, UriKind.Absolute) Then
            uri = New Uri(Location, UriKind.Absolute)
        Else
            Dim HostUri As String = Response.ResponseUri.GetComponents(UriComponents.SchemeAndServer,
                                                                        UriFormat.Unescaped) + Location
            uri = If(uri.IsWellFormedUriString(HostUri, UriKind.Absolute),
                        New Uri(HostUri),
                        New Uri(Response.ResponseUri.GetComponents(UriComponents.Scheme, UriFormat.Unescaped) +
                                Response.ResponseUri.Host + Location))
        End If
    Catch ExceptionOnInvalidUri As Exception
        uri = New Uri(Location, UriKind.Relative)
    End Try

    Return uri

End Function

Private Function CertificateValidation(sender As Object,
                                        CACert As X509Certificate,
                                        CAChain As X509Chain,
                                        PolicyErrors As SslPolicyErrors) As Boolean

    'This method, as it is, accepts a Server certificate in any case
    'It could be eventually adapted to refuse a connection (returning false) 
    'if the certificate is invalid, expired or from a untrusted path
    If (PolicyErrors = SslPolicyErrors.None) Then Return True

    'If a Certificated must be added to the Chain, uncomment the code below,
    'selecting a Certificate in the Local (or other) Storage
    'Dim MyCert As X509Certificate2 = New X509Certificate2("[localstorage]/[ca.cert]")
    'CAChain.ChainPolicy.ExtraStore.Add(MyCert)

    'CAChain.Build(MyCert)
    'For Each CACStatus As X509ChainStatus In CAChain.ChainStatus
    '    If (CACStatus.Status <> X509ChainStatusFlags.NoError) And
    '       (CACStatus.Status <> X509ChainStatusFlags.UntrustedRoot) Then
    '        Return False
    '    End If
    'Next

    Return True

End Function