从SslStream读取响应时不能信任Content-Length吗？

Question

在.NET Core 2.2上玩TcpClient和NetworkStream。
尝试从https://www.google.com/

获取内容

在继续之前，我想说明一下，我不想使用WebClient，HttpWebRequest或HttpClient类。在很多问题中，人们在使用TcpClient时遇到了一些问题，而响应者或评论者建议在此任务中使用其他方式，所以请不要。

假设我们有一个从TcpClient的NetworkStream获得并经过正确身份验证的SslStream实例。

让我们假设还有一个StreamWriter用于将HTTP消息写入此流，还有一个StreamReader用于从响应中读取HTTP消息头：

var tcpClient = new TcpClient("google.com", 443);
var stream = tcpClient.GetStream();
var sslStream = new SslStream(stream, false);
sslStream.AuthenticateAsClient("google.com");
var streamWriter = new StreamWriter(sslStream);
var streamReader = new StreamReader(sslStream);

假设我们发送请求的方式与Firefox浏览器发送请求的方式相同：

GET / HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: sr,sr-RS;q=0.8,sr-CS;q=0.6,en-US;q=0.4,en;q=0.2
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0

这将导致发送以下响应：

HTTP/1.1 200 OK
Date: Sun, 28 Apr 2019 17:28:27 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=UTF-8
Strict-Transport-Security: max-age=31536000
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Content-Encoding: br
Server: gws
Content-Length: 55786
... etc

现在，在使用streamReader.ReadLine()读取所有响应标头并解析了响应标头中的内容长度之后，让我们将响应内容读入缓冲区：

var totalBytesRead = 0;
int bytesRead;
var buffer = new byte[contentLength];
do
{
    bytesRead = sslStream.Read(buffer,
        totalBytesRead,
        contentLength - totalBytesRead);
    totalBytesRead += bytesRead;
} while (totalBytesRead < contentLength && bytesRead > 0);

但是，此do..while循环仅在远程服务器关闭连接后退出，这意味着对Read的最后一次调用将挂起。这意味着我们已经阅读了整个响应内容，并且服务器已经在侦听此流上的另一个HTTP消息。 contentLength不正确吗？调用streamReader时ReadLine是否读得太多，是否会使SslStream位置混乱，从而导致读取无效数据？

有什么作用？有人有经验吗？

P.S。这是一个示例控制台应用程序代码，其中省略了所有安全检查，从而证明了这一点：

private static void Main(string[] args)
{
    using (var tcpClient = new TcpClient("google.com", 443))
    {
        var stream = tcpClient.GetStream();
        using (var sslStream = new SslStream(stream, false))
        {
            sslStream.AuthenticateAsClient("google.com");
            using (var streamReader = new StreamReader(sslStream))
            using (var streamWriter = new StreamWriter(sslStream))
            {
                streamWriter.WriteLine("GET / HTTP/1.1");
                streamWriter.WriteLine("Host: www.google.com");
                streamWriter.WriteLine("User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
                streamWriter.WriteLine("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
                streamWriter.WriteLine("Accept-Language: sr,sr-RS;q=0.8,sr-CS;q=0.6,en-US;q=0.4,en;q=0.2");
                streamWriter.WriteLine("Accept-Encoding: gzip, deflate, br");
                streamWriter.WriteLine("Connection: keep-alive");
                streamWriter.WriteLine("Upgrade-Insecure-Requests: 1");
                streamWriter.WriteLine("Cache-Control: max-age=0");
                streamWriter.WriteLine();
                streamWriter.Flush();

                var lines = new List<string>();
                var line = streamReader.ReadLine();
                var contentLength = 0;
                while (!string.IsNullOrWhiteSpace(line))
                {
                    var split = line.Split(": ");
                    if (split.First() == "Content-Length")
                    {
                        contentLength = int.Parse(split[1]);
                    }

                    lines.Add(line);
                    line = streamReader.ReadLine();
                }

                var totalBytesRead = 0;
                int bytesRead;
                var buffer = new byte[contentLength];
                do
                {
                    bytesRead = sslStream.Read(buffer,
                        totalBytesRead,
                        contentLength - totalBytesRead);
                    totalBytesRead += bytesRead;
                    Console.WriteLine(
                        $"Bytes read: {totalBytesRead} of {contentLength} (last chunk: {bytesRead} bytes)");
                } while (totalBytesRead < contentLength && bytesRead > 0);

                Console.WriteLine(
                    "--------------------");
            }
        }
    }

    Console.ReadLine();
}

编辑

这总是在我提交问题后发生。我已经花了几天的时间挠头，却找不到问题的原因，但是当我提交问题后，我就知道这与StreamReader试图弄乱事情有关读一行。

因此，如果我停止使用StreamReader并将对ReadLine的调用替换为逐字节读取的内容，那么一切似乎都很好。替换代码可以编写如下：

private static IEnumerable<string> ReadHeader(Stream sslStream)
{
    // One-byte buffer for reading bytes from the stream
    var buffer = new byte[1];

    // Initialize a four-character string to keep the last four bytes of the message
    var check = new StringBuilder("....");
    int bytes;
    var responseBuilder = new StringBuilder();
    do
    {
        // Read the next byte from the stream and write in into the buffer
        bytes = sslStream.Read(buffer, 0, 1);
        if (bytes == 0)
        {
            // If nothing was read, break the loop
            break;
        }

        // Add the received byte to the response builder.
        // We expect the header to be ASCII encoded so it's OK to just cast to char and append
        responseBuilder.Append((char) buffer[0]);

        // Always remove the first char from the string and append the latest received one
        check.Remove(0, 1);
        check.Append((char) buffer[0]);

        // \r\n\r\n marks the end of the message header, so break here
        if (check.ToString() == "\r\n\r\n")
        {
            break;
        }
    } while (bytes > 0);

    var headerText = responseBuilder.ToString();
    return headerText.Split("\r\n", StringSplitOptions.RemoveEmptyEntries);
}

...这将使我们的示例控制台应用看起来像这样：

private static void Main(string[] args)
{
    using (var tcpClient = new TcpClient("google.com", 443))
    {
        var stream = tcpClient.GetStream();
        using (var sslStream = new SslStream(stream, false))
        {
            sslStream.AuthenticateAsClient("google.com");
            using (var streamWriter = new StreamWriter(sslStream))
            {
                streamWriter.WriteLine("GET / HTTP/1.1");
                streamWriter.WriteLine("Host: www.google.com");
                streamWriter.WriteLine("User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
                streamWriter.WriteLine("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
                streamWriter.WriteLine("Accept-Language: sr,sr-RS;q=0.8,sr-CS;q=0.6,en-US;q=0.4,en;q=0.2");
                streamWriter.WriteLine("Accept-Encoding: gzip, deflate, br");
                streamWriter.WriteLine("Connection: keep-alive");
                streamWriter.WriteLine("Upgrade-Insecure-Requests: 1");
                streamWriter.WriteLine("Cache-Control: max-age=0");
                streamWriter.WriteLine();
                streamWriter.Flush();

                var lines = ReadHeader(sslStream);
                var contentLengthLine = lines.First(x => x.StartsWith("Content-Length"));
                var split = contentLengthLine.Split(": ");
                var contentLength = int.Parse(split[1]);

                var totalBytesRead = 0;
                int bytesRead;
                var buffer = new byte[contentLength];
                do
                {
                    bytesRead = sslStream.Read(buffer,
                        totalBytesRead,
                        contentLength - totalBytesRead);
                    totalBytesRead += bytesRead;
                    Console.WriteLine(
                        $"Bytes read: {totalBytesRead} of {contentLength} (last chunk: {bytesRead} bytes)");
                } while (totalBytesRead < contentLength && bytesRead > 0);

                Console.WriteLine(
                    "--------------------");
            }
        }
    }

    Console.ReadLine();
}

private static IEnumerable<string> ReadHeader(Stream sslStream)
{
    // One-byte buffer for reading bytes from the stream
    var buffer = new byte[1];

    // Initialize a four-character string to keep the last four bytes of the message
    var check = new StringBuilder("....");
    int bytes;
    var responseBuilder = new StringBuilder();
    do
    {
        // Read the next byte from the stream and write in into the buffer
        bytes = sslStream.Read(buffer, 0, 1);
        if (bytes == 0)
        {
            // If nothing was read, break the loop
            break;
        }

        // Add the received byte to the response builder.
        // We expect the header to be ASCII encoded so it's OK to just cast to char and append
        responseBuilder.Append((char)buffer[0]);

        // Always remove the first char from the string and append the latest received one
        check.Remove(0, 1);
        check.Append((char)buffer[0]);

        // \r\n\r\n marks the end of the message header, so break here
        if (check.ToString() == "\r\n\r\n")
        {
            break;
        }
    } while (bytes > 0);

    var headerText = responseBuilder.ToString();
    return headerText.Split("\r\n", StringSplitOptions.RemoveEmptyEntries);
}

Answer 1

标题中问题的答案为是。
只要您正确阅读了邮件标题，即不使用StreamReader.ReadLine，就可以信任它。

这是完成任务的实用程序方法：

private static string ReadStreamUntil(Stream stream, string boundary)
{
    // One-byte buffer for reading bytes from the stream
    var buffer = new byte[1];

    // Initialize a string builder with some placeholder chars of the length as the boundary
    var boundaryPlaceholder = string.Join(string.Empty, boundary.Select(x => "."));
    var check = new StringBuilder(boundaryPlaceholder);
    var responseBuilder = new StringBuilder();
    do
    {
        // Read the next byte from the stream and write in into the buffer
        var byteCount = stream.Read(buffer, 0, 1);
        if (byteCount == 0)
        {
            // If nothing was read, break the loop
            break;
        }

        // Add the received byte to the response builder.
        responseBuilder.Append((char)buffer[0]);

        // Always remove the first char from the string and append the latest received one
        check.Remove(0, 1);
        check.Append((char)buffer[0]);

        // boundary marks the end of the message, so break here
    } while (check.ToString() != boundary);

    return responseBuilder.ToString();
}

然后，要读取标题，我们只需调用ReadStreamUntil(sslStream, "\r\n\r\n")。

这里的关键是逐字节读取流，直到遇到已知的字节序列（在这种情况下为\ r \ n \ r \ n）。

使用此方法读取后，流将位于正确位置，以便正确读取响应内容。

如果有什么好处，可以通过调用await ReadAsync而不是Read轻松地将该方法转换为异步变量。

值得注意的是，上述方法仅在文本是ASCII编码的情况下才能正常工作。