Question

我正在使用HttpWebRequest通过POST方法获取网页源代码。需要使用不同的参数多次访问页面。

在短时间内第二次获得响应时，它总是向我返回相同的响应对象。调试后，如果第二次调用大约30秒后，请求可以获得正确的响应对象（源代码）。

public HtmlAgilityPack.HtmlDocument getHtmlData(string url, string cname)
    {
        HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();

        try
        {
            HttpRequestCachePolicy policy = new HttpRequestCachePolicy(HttpRequestCacheLevel.Default);
            HttpWebRequest.DefaultCachePolicy = policy;
            HttpRequestCachePolicy noCachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore);

            HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;
            req.Method = "POST";
            req.ContentType = "application/x-www-form-urlencoded";
            req.KeepAlive = false;
            req.Headers[HttpRequestHeader.CacheControl] = "no-cache";
            req.Headers[HttpRequestHeader.Pragma] = "no-cache";
            req.IfModifiedSince = DateTime.Now;
            req.CachePolicy = noCachePolicy;

            //add post data
            string postData = "cname=" + cname;
            byte[] byteArray = Encoding.GetEncoding("shift-jis").GetBytes(postData);
            req.ContentLength = byteArray.Length;
            using (Stream stream = req.GetRequestStream())
            {
                stream.Write(byteArray, 0, byteArray.Length);
            }

            // get response
            string data = "";
            HttpWebResponse response = req.GetResponse() as HttpWebResponse;
            using (var stream = response.GetResponseStream())
            {
                StreamReader reader = new StreamReader(stream, Encoding.GetEncoding("shift-jis"));
                data = reader.ReadToEnd();
                reader.Close();
            }

            response.Close();
            doc.LoadHtml(data);
            //System.Threading.Thread.Sleep(30000);
        }
        catch(Exception ex)
        {
            Log.log.Debug(ex.ToString());
        }

        return doc;
    }

函数调用块：

getHtmlData(@"http://www.jra.go.jp/JRADB/accessS.html", "pw01sli00/AF");
getHtmlData(@"http://www.jra.go.jp/JRADB/accessS.html", "pw01skl00999999/B3");
getHtmlData(@"http://www.jra.go.jp/JRADB/accessS.html", "pw01skl00201604/E3");

我一直坚持这个问题一整天。希望有人能给我一些线索。

非常感谢！

HttpWebRequest在第二次调用时获取相同的响应对象

0 个答案: