我正在尝试使用多个线程进行webrequests,但如果我尝试使用超过2个,我会收到错误
Index was outside the bonds of the array
在这一行:
string username = ScrapeBox1.Lines[NamesCounter].ToString();
以下是代码:
while (working)
{
while (usernamescount > NamesCounter)
{
string username = ScrapeBox1.Lines[NamesCounter].ToString();
string url = "http://www.someforum.com/members/" + username + ".html";
var request = (HttpWebRequest)(WebRequest.Create(url));
var response = request.GetResponse();
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0";
using (var responseStream = response.GetResponseStream())
{
using (var responseStreamReader = new StreamReader(responseStream))
{
var serverResponse = responseStreamReader.ReadToEnd();
int startpoint = serverResponse.IndexOf("Contact Info</span>");
try
{
string strippedResponse = serverResponse.Remove(0, startpoint);
ExtractEmails(strippedResponse);
}
catch { }
}
}
NamesCounter++;
textBox1.Text = NamesCounter.ToString();
}
}
答案 0 :(得分:2)
此代码不是线程安全的。
您需要用于执行HttpWebRequest的代码是原子的,并且在循环遍历集合的上下文之外。
例如
public void MakeHttpWebRequest(string userName)
{
string url = "http://www.someforum.com/members/" + userName + ".html";
var request = (HttpWebRequest)(WebRequest.Create(url));
var response = request.GetResponse();
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0";
using (var responseStream = response.GetResponseStream())
{
using (var responseStreamReader = new StreamReader(responseStream))
{
var serverResponse = responseStreamReader.ReadToEnd();
int startpoint = serverResponse.IndexOf("Contact Info</span>");
try
{
string strippedResponse = serverResponse.Remove(0, startpoint);
ExtractEmails(strippedResponse);
}
catch { }
}
}
}
假设ScrapeBox.Lines实现了IEnumerable,我建议使用Parallel.ForEach并将ScrapeBox.Lines作为IEnumerable传递给它进行迭代。
现在,还有一个问题,用于从HttpWebRequest读取响应的代码仍然需要将其输出写入共享位置。以线程安全的方式实现这一点。执行此操作的常用方法是使用信号量。您需要一个可供每个线程实例访问的对象。类级私有变量private object sharedMutex = new object();
可以工作。然后应将代码ExtractEmails(strippedResponse);
更改为
lock(sharedMutex)
{
ExtractEmails(strippedResponse);
}
如果没有ExtractEmails(<string>)
方法的代码,我就无法为此提供线程安全的实现,因此解决方案的某些部分仍可能导致问题。