Question

我正在研究Google Cloud Storage .NET client library。有三个功能（.NET，我的客户端之间）库中的库和存储服务组合在一起不愉快的方式：

下载文件（Google云端存储中的对象）术语），服务器包括存储数据的散列。我的客户端代码然后根据它的数据验证该哈希值下载。
Google云端存储的一个独特功能是用户可以设置对象的Content-Encoding，并将其包含在内下载时的标题，当请求包含匹配时接受编码。（目前，让我们忽略时的行为请求不包括......）
HttpClientHandler可以解压缩gzip（或deflate）内容自动透明。

当所有这三者结合在一起时，我们就会遇到麻烦。这是一个简短但完整的程序证明了这一点，但没有使用我的客户端库（并点击可公开访问的文件）：

using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        string url = "https://www.googleapis.com/download/storage/v1/b/"
            + "storage-library-test-bucket/o/gzipped-text.txt?alt=media";
        var handler = new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.GZip
        };
        var client = new HttpClient(handler);

        var response = await client.GetAsync(url);
        byte[] content = await response.Content.ReadAsByteArrayAsync();
        string text = Encoding.UTF8.GetString(content);
        Console.WriteLine($"Content: {text}");

        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");

        using (var md5 = MD5.Create())
        {
            var md5Hash = md5.ComputeHash(content);
            var md5HashBase64 = Convert.ToBase64String(md5Hash);
            Console.WriteLine($"MD5 of content: {md5HashBase64}");
        }
    }
}

.NET Core项目文件：

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <LangVersion>7.1</LangVersion>
  </PropertyGroup>
</Project>

输出：

Content: hello world
Hash header: crc32c=T1s5RQ==,md5=xhF4M6pNFRDQnvaRRNVnkA==
MD5 of content: XrY7u+Ae7tCTyyK7j1rNww==

如您所见，内容的MD5与MD5不同 X-Goog-Hash标题的一部分。（在我的客户端库中，我正在使用crc32c hash，但显示相同的行为。）

这不是HttpClientHandler中的错误 - 这是预期的，但是很痛苦当我想验证哈希。基本上，我需要在解压缩后和之前的内容。我找不到任何办法这样做。

为了澄清我的要求，我知道如何防止HttpClient中的解压缩，而是在从流中读取后解压缩 - 但我需要能够在不更改任何使用结果的代码的情况下执行此操作来自HttpResponseMessage的{{1}}。（有很多代码可以处理响应，我只希望在一个中心位置进行更改。）

我有一个计划，我已经制作了原型，并且我的工作正常发现到目前为止，但有点难看。它涉及创建一个三层处理程序：

HttpClient已停用自动解压缩功能。
一个新的处理程序，用新的HttpClientHandler子类替换内容流它委托给原始内容流，但在读取时散列数据。
基于Microsoft DecompressionHandler代码的仅解压缩处理程序。

虽然这有效，但它有以下缺点：

开源许可：按顺序检查我需要做什么根据MIT许可在我的仓库中创建一个新文件 Microsoft代码
有效地分发MS代码，这意味着我应该定期检查是否发现了任何错误
Microsoft代码使用程序集的内部成员，因此它没有尽可能干净地移植。

如果微软公开Stream，那将有助于很多 - 但这可能比我需要的时间更长。

如果可能的话，我正在寻找的是另一种方法 - 我错过的东西让我以前得到的内容减压。我不想重新发明DecompressionHandler - 回复例如，经常是分块，我不想进入事情的那一面。这是一个非常具体的拦截点我正在寻找。

Answer 1

看看@Michael给我的提示我失踪了。获取压缩内容后，您可以使用CryptoStream，GZipStream和StreamReader来阅读响应，而无需将其加载到内存中。 CryptoStream会在解压缩并读取压缩内容时对其进行哈希处理。将StreamReader替换为FileStream，您可以将数据写入内存使用量最少的文件中：）

using System;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        string url = "https://www.googleapis.com/download/storage/v1/b/"
            + "storage-library-test-bucket/o/gzipped-text.txt?alt=media";
        var handler = new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.None
        };
        var client = new HttpClient(handler);
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");

        var response = await client.GetAsync(url);
        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");
        string text = null;
        using (var md5 = MD5.Create())
        {
            using (var cryptoStream = new CryptoStream(await response.Content.ReadAsStreamAsync(), md5, CryptoStreamMode.Read))
            {
                using (var gzipStream = new GZipStream(cryptoStream, CompressionMode.Decompress))
                {
                    using (var streamReader = new StreamReader(gzipStream, Encoding.UTF8))
                    {
                        text = streamReader.ReadToEnd();
                    }
                }
                Console.WriteLine($"Content: {text}");
                var md5HashBase64 = Convert.ToBase64String(md5.Hash);
                Console.WriteLine($"MD5 of content: {md5HashBase64}");
            }
        }
    }
}

输出：

Hash header: crc32c=T1s5RQ==,md5=xhF4M6pNFRDQnvaRRNVnkA==
Content: hello world
MD5 of content: xhF4M6pNFRDQnvaRRNVnkA==

答案的V2

在阅读Jon的回复和更新的答案后，我有以下版本。几乎相同的想法，但我将流媒体移动到我注入的特殊HttpContent。不完全漂亮，但想法就在那里。

using System;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        string url = "https://www.googleapis.com/download/storage/v1/b/"
            + "storage-library-test-bucket/o/gzipped-text.txt?alt=media";
        var handler = new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.None
        };
        var client = new HttpClient(new Intercepter(handler));
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");

        var response = await client.GetAsync(url);
        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");
        HttpContent content1 = response.Content;
        byte[] content = await content1.ReadAsByteArrayAsync();
        string text = Encoding.UTF8.GetString(content);
        Console.WriteLine($"Content: {text}");
        var md5Hash = ((HashingContent)content1).Hash;
        var md5HashBase64 = Convert.ToBase64String(md5Hash);
        Console.WriteLine($"MD5 of content: {md5HashBase64}");
    }

    public class Intercepter : DelegatingHandler
    {
        public Intercepter(HttpMessageHandler innerHandler) : base(innerHandler)
        {
        }

        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            var response = await base.SendAsync(request, cancellationToken);
            response.Content = new HashingContent(await response.Content.ReadAsStreamAsync());
            return response;
        }
    }

    public sealed class HashingContent : HttpContent
    {
        private readonly StreamContent streamContent;
        private readonly MD5 mD5;
        private readonly CryptoStream cryptoStream;
        private readonly GZipStream gZipStream;

        public HashingContent(Stream content)
        {
            mD5 = MD5.Create();
            cryptoStream = new CryptoStream(content, mD5, CryptoStreamMode.Read);
            gZipStream = new GZipStream(cryptoStream, CompressionMode.Decompress);
            streamContent = new StreamContent(gZipStream);
        }

        protected override Task SerializeToStreamAsync(Stream stream, TransportContext context) => streamContent.CopyToAsync(stream, context);
        protected override bool TryComputeLength(out long length)
        {
            length = 0;
            return false;
        }

        protected override Task<Stream> CreateContentReadStreamAsync() => streamContent.ReadAsStreamAsync();

        protected override void Dispose(bool disposing)
        {
            try
            {
                if (disposing)
                {
                    streamContent.Dispose();
                    gZipStream.Dispose();
                    cryptoStream.Dispose();
                    mD5.Dispose();
                }
            }
            finally
            {
                base.Dispose(disposing);
            }
        }

        public byte[] Hash => mD5.Hash;
    }
}

Answer 2

我设法通过以下方式使标题正确：

创建一个继承HttpClientHandler的自定义处理程序
覆盖SendAsync
使用base.SendAsync
使用GZipStream
将Gzip Md5哈希到base64（使用您的代码）

这个问题就像你说的那样＃34;在减压之前＃34;这里并没有真正受到尊重

我们的想法是让if按照您的意愿运作 https://github.com/dotnet/corefx/blob/master/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseParser.cs#L80-L91

匹配

class Program
{
    const string url = "https://www.googleapis.com/download/storage/v1/b/storage-library-test-bucket/o/gzipped-text.txt?alt=media";

    static async Task Main()
    {
        //await HashResponseContent(CreateHandler(DecompressionMethods.None));
        //await HashResponseContent(CreateHandler(DecompressionMethods.GZip));
        await HashResponseContent(new MyHandler());

        Console.ReadLine();
    }

    private static HttpClientHandler CreateHandler(DecompressionMethods decompressionMethods)
    {
        return new HttpClientHandler { AutomaticDecompression = decompressionMethods };
    }

    public static async Task HashResponseContent(HttpClientHandler handler)
    {
        //Console.WriteLine($"Using AutomaticDecompression : '{handler.AutomaticDecompression}'");
        //Console.WriteLine($"Using SupportsAutomaticDecompression : '{handler.SupportsAutomaticDecompression}'");
        //Console.WriteLine($"Using Properties : '{string.Join('\n', handler.Properties.Keys.ToArray())}'");

        var client = new HttpClient(handler);

        var response = await client.GetAsync(url);
        byte[] content = await response.Content.ReadAsByteArrayAsync();
        string text = Encoding.UTF8.GetString(content);
        Console.WriteLine($"Content: {text}");

        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");
        byteArrayToMd5(content);

        Console.WriteLine($"=====================================================================");
    }

    public static string byteArrayToMd5(byte[] content)
    {
        using (var md5 = MD5.Create())
        {
            var md5Hash = md5.ComputeHash(content);
            return Convert.ToBase64String(md5Hash);
        }
    }

    public static byte[] Compress(byte[] contentToGzip)
    {
        using (MemoryStream resultStream = new MemoryStream())
        {
            using (MemoryStream contentStreamToGzip = new MemoryStream(contentToGzip))
            {
                using (GZipStream compressionStream = new GZipStream(resultStream, CompressionMode.Compress))
                {
                    contentStreamToGzip.CopyTo(compressionStream);
                }
            }

            return resultStream.ToArray();
        }
    }
}

public class MyHandler : HttpClientHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var response = await base.SendAsync(request, cancellationToken);
        var responseContent = await response.Content.ReadAsByteArrayAsync().ConfigureAwait(false);

        Program.byteArrayToMd5(responseContent);

        var compressedResponse = Program.Compress(responseContent);
        var compressedResponseMd5 = Program.byteArrayToMd5(compressedResponse);

        Console.WriteLine($"recompressed response to md5 : {compressedResponseMd5}");

        return response;
    }
}

Answer 3

如何禁用自动解压缩，手动添加Accept-Encoding标头，然后在哈希验证后解压缩？

private static async Task Test2()
{
    var url = @"https://www.googleapis.com/download/storage/v1/b/storage-library-test-bucket/o/gzipped-text.txt?alt=media";
    var handler = new HttpClientHandler
    {
        AutomaticDecompression = DecompressionMethods.None
    };
    var client = new HttpClient(handler);
    client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");

    var response = await client.GetAsync(url);
    var raw = await response.Content.ReadAsByteArrayAsync();

    var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
    Debug.WriteLine($"Hash header: {hashHeader}");

    bool match = false;
    using (var md5 = MD5.Create())
    {
        var md5Hash = md5.ComputeHash(raw);
        var md5HashBase64 = Convert.ToBase64String(md5Hash);
        match = hashHeader.EndsWith(md5HashBase64);
        Debug.WriteLine($"MD5 of content: {md5HashBase64}");
    }

    if (match)
    {
        var memInput = new MemoryStream(raw);
        var gz = new GZipStream(memInput, CompressionMode.Decompress);
        var memOutput = new MemoryStream();
        gz.CopyTo(memOutput);
        var text = Encoding.UTF8.GetString(memOutput.ToArray());
        Console.WriteLine($"Content: {text}");
    }
}

是否可以在HttpClient中解压缩之前访问压缩数据？

3 个答案: