Question

我写了一个从网站获取html的类，这是代码：

public class NetworkHelper {
    static Lazy<HttpClient> httpClient = new Lazy<HttpClient>(() => {
        var handler = CreateHandler();
        return new HttpClient(handler) {
            Timeout = TimeSpan.FromSeconds(3)
        };
    });

    static HttpMessageHandler CreateHandler() {
        var handler = new HttpClientHandler();
        // if the framework supports redirect configuration
        // set max redirect to the desired amount the default is 50
        if (handler.SupportsRedirectConfiguration) {
            handler.AllowAutoRedirect = true;
            handler.MaxAutomaticRedirections = 5;
        }
        // if the framework supports automatic decompression 
        // set automatic decompression
        if (handler.SupportsAutomaticDecompression) {
            handler.AutomaticDecompression = System.Net.DecompressionMethods.GZip |
                System.Net.DecompressionMethods.Deflate;
        }
        return handler;
    }

    /// <summary>
    /// Get the html structure of a site.
    /// </summary>
    /// <param name="url">Represents the URL of the page where to download the data.</param>
    /// <returns>Return a string that contains the html of the site.</returns>
    public async Task<string> GetHtmlAsync(Uri url, CancellationToken cancellationToken = default(CancellationToken)) {
        var response = await httpClient.Value.GetAsync(url, cancellationToken);
        var content = await response.Content.ReadAsStringAsync();
        return content;
    }
}

问题在于，当我尝试从此网址下载数据时：

string html = await new NetworkHelper().GetHtmlAsync(new Uri("https://int.soccerway.com/charts/statsplus/2139109"));

我会得到这样的内容：

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://int.soccerway.com/charts/statsplus/2139109/">here</a>.</p>
</body></html>

所以我猜AllowAutoRedirect没有按预期工作？

Answer 1

我发现改变了：

return new HttpClient(handler) {
        Timeout = TimeSpan.FromSeconds(3)
    };

为：

return new HttpClient(handler) {
            //Timeout = TimeSpan.FromSeconds(3)
        };

一切顺利。似乎每秒可以执行的HTTP请求数量有限，其他请求排队。

301使用AllowAutoRedirect永久移动

1 个答案: