我正在线程化以下函数,当我运行超过10个线程时,我开始获得一堆超时连接..
我已将_wReq.ServicePoint.ConnectionLimit = 100;
添加到方法中并添加了
以下是我的app.config
<system.net>
<connectionManagement>
<add address = "*" maxconnection = "200" />
</connectionManagement>
</system.net>
以下是方法:
public string websocket(string proxy, string URL)
{
string _html = "";
HttpWebResponse _wResp = null;
try
{
HttpWebRequest _wReq;
System.Text.ASCIIEncoding _enc = new System.Text.ASCIIEncoding();
_wReq = (HttpWebRequest)HttpWebRequest.Create(URL);
_wReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
_wReq.Accept = "*/*";
_wReq.Method = "GET";
_wReq.CookieContainer = cookieJarf;
_wReq.ServicePoint.ConnectionLimit = 100;
_wReq.KeepAlive = false;
_wReq.Timeout = 5000;
_wReq.ReadWriteTimeout = 5000;
_wReq.Pipelined = true;
if (proxy != "")
{
WebProxy myproxy = new WebProxy(proxy);
_wReq.Proxy = myproxy;
}
_wResp = (HttpWebResponse)_wReq.GetResponse();
Encoding responseEncoding = Encoding.GetEncoding(_wResp.CharacterSet);
using (StreamReader sr = new StreamReader(_wResp.GetResponseStream(), responseEncoding))
{
_html = sr.ReadToEnd();
}
cookieJarf = _wReq.CookieContainer;
_wResp.Close();
}
catch (WebException wexc1)
{
MessageBox.Show(wexc1.Message); // Time out exception when running more than 10 threads
if (wexc1.Status == WebExceptionStatus.ProtocolError)
{
return "";
}
}
finally
{
if (_wResp != null)
_wResp.Close();
}
return _html;
}
我并不是出于我应该尝试的想法:/当然,必须能够同时运行50个httpwebrequests?
编辑:应该说这些线程连接中的一些是同一个服务器而另一些是不同的。我基本上都是使用这种方法进行线程爬虫。
编辑1:
点击按钮后添加我的线程:
foreach (string item in urlQueue)
{
_smartThreadPool.QueueWorkItem(
new Amib.Threading.Func<string, int, int, string, int>(checkURL),
item, iia, 5000, kryptonTextBox1.Text);
iia++;
}