Question

您好我正在运行一个简单的HTML抓取工具，但是在某些网站上（大约10％）我收到Cookie解析错误：

System.Net.CookieException：解析Uri的Cookie标头时发生错误

System.ArgumentOutOfRangeException：Count必须为正数，count必须引用string / array / collection中的某个位置。参数名称：count

守则

HttpWebRequest myHttpWebRequest = 
(HttpWebRequest)WebRequest.Create(getURLtoScrape);

myHttpWebRequest.Method = "GET";
myHttpWebRequest.Timeout = 10000; // 10 secs
myHttpWebRequest.MaximumResponseHeadersLength = 10;

myHttpWebRequest.UserAgent = userAgent;

myHttpWebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
myHttpWebRequest.Headers.Set("Accept-Language", "en-US,en;q=0.8");

CookieContainer cookies = new CookieContainer();
myHttpWebRequest.CookieContainer = cookies;

myHttpWebRequest.MaximumAutomaticRedirections = 5;
myHttpWebRequest.AllowAutoRedirect = true;

HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();


string resolveURL = Convert.ToString(myHttpWebResponse.ResponseUri);
Console.WriteLine(">> resolveURL:" + resolveURL + "");

Stream streamResponse = myHttpWebResponse.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
Char[] readBuffer = new Char[256];
int count = streamRead.Read(readBuffer, 0, 256);


while (count > 0)
{
    String outputData = new String(readBuffer, 0, count);
    html = html + outputData;
    count = streamRead.Read(readBuffer, 0, 256);
}

streamRead.Close();
streamResponse.Close();
myHttpWebResponse.Close();

错误：

System.AggregateException: One or more errors occurred. ---> System.Net.CookieException: An error occurred when parsing the Cookie header for Uri 'http://www.curriechevy.com/'. ---> System.ArgumentOutOfRangeException: Count must be positive and count must refer to a location within the string/array/collection.
Parameter name: count
   at System.String.IndexOf(Char value, Int32 startIndex, Int32 count)
   at System.Net.Cookie.VerifySetDefaults(CookieVariant variant, Uri uri, Boolean isLocalDomain, String localDomain, Boolean set_default, Boolean isThrow)
   at System.Net.CookieContainer.CookieCutter(Uri uri, String headerName, String setCookieHeader, Boolean isThrow)
   --- End of inner exception stack trace ---
   at System.Net.CookieContainer.CookieCutter(Uri uri, String headerName, String setCookieHeader, Boolean isThrow)
   at System.Net.CookieContainer.SetCookies(Uri uri, String cookieHeader)
   at ScrapySharp.Network.ScrapingBrowser.<GetWebResponseAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ScrapySharp.Network.ScrapingBrowser.<GetResponseAsync>d__49.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ScrapySharp.Network.ScrapingBrowser.<NavigateToPageAsync>d__64.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at ScrapySharp.Network.ScrapingBrowser.NavigateToPage(Uri url, HttpVerb verb, String data, String contentType)
   at pageScrape.Program.Main(String[] args) in d:\Corporate\AutoTrader\ConsoleApplications\AutoWebSite-CompetitorScrape\ConsoleApplication1\Program.cs:line 267
---> (Inner Exception #0) System.Net.CookieException: An error occurred when parsing the Cookie header for Uri 'http://www.curriechevy.com/'. ---> System.ArgumentOutOfRangeException: Count must be positive and count must refer to a location within the string/array/collection.
Parameter name: count
   at System.String.IndexOf(Char value, Int32 startIndex, Int32 count)
   at System.Net.Cookie.VerifySetDefaults(CookieVariant variant, Uri uri, Boolean isLocalDomain, String localDomain, Boolean set_default, Boolean isThrow)
   at System.Net.CookieContainer.CookieCutter(Uri uri, String headerName, String setCookieHeader, Boolean isThrow)
   --- End of inner exception stack trace ---
   at System.Net.CookieContainer.CookieCutter(Uri uri, String headerName, String setCookieHeader, Boolean isThrow)
   at System.Net.CookieContainer.SetCookies(Uri uri, String cookieHeader)
   at ScrapySharp.Network.ScrapingBrowser.<GetWebResponseAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ScrapySharp.Network.ScrapingBrowser.<GetResponseAsync>d__49.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ScrapySharp.Network.ScrapingBrowser.<NavigateToPageAsync>d__64.MoveNext()<---

Web scraper抛出零星的System.Net.CookieException System.ArgumentOutOfRangeException

0 个答案: