Httpwebrequest + threading =超时?

时间:2012-10-10 13:04:58

标签: c# multithreading httpwebrequest

我正在线程化以下函数,当我运行超过10个线程时,我开始获得一堆超时连接..

我已将_wReq.ServicePoint.ConnectionLimit = 100;添加到方法中并添加了 以下是我的app.config

<system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "200" />
    </connectionManagement>
</system.net>

以下是方法:

public string websocket(string proxy, string URL)
{
    string _html = "";
    HttpWebResponse _wResp = null;

    try
    {
        HttpWebRequest _wReq;
        System.Text.ASCIIEncoding _enc = new System.Text.ASCIIEncoding();
        _wReq = (HttpWebRequest)HttpWebRequest.Create(URL);
        _wReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
        _wReq.Accept = "*/*";
        _wReq.Method = "GET";
        _wReq.CookieContainer = cookieJarf;
        _wReq.ServicePoint.ConnectionLimit = 100;
        _wReq.KeepAlive = false;
        _wReq.Timeout = 5000;
        _wReq.ReadWriteTimeout = 5000;
        _wReq.Pipelined = true;

        if (proxy != "")
        {
            WebProxy myproxy = new WebProxy(proxy);
            _wReq.Proxy = myproxy;
        }

        _wResp = (HttpWebResponse)_wReq.GetResponse();
        Encoding responseEncoding = Encoding.GetEncoding(_wResp.CharacterSet);
        using (StreamReader sr = new StreamReader(_wResp.GetResponseStream(), responseEncoding))
        {
            _html = sr.ReadToEnd();
        }

        cookieJarf = _wReq.CookieContainer;
        _wResp.Close();
    }
    catch (WebException wexc1)
    {
        MessageBox.Show(wexc1.Message); // Time out exception when running more than 10 threads

        if (wexc1.Status == WebExceptionStatus.ProtocolError)
        {
            return "";
        }
    }
    finally
    {
         if (_wResp != null)
             _wResp.Close();
    }
    return _html;
}

我并不是出于我应该尝试的想法:/当然,必须能够同时运行50个httpwebrequests?

编辑:应该说这些线程连接中的一些是同一个服务器而另一些是不同的。我基本上都是使用这种方法进行线程爬虫。

编辑1:

点击按钮后添加我的线程:

foreach (string item in urlQueue)
{
    _smartThreadPool.QueueWorkItem(
    new Amib.Threading.Func<string, int, int, string, int>(checkURL),
    item, iia, 5000, kryptonTextBox1.Text);

    iia++;
}

0 个答案:

没有答案