Question

我有一个组件，它在每个单独的线程中处理多个Web请求。每个WebRequest处理都是同步的。

public class WebRequestProcessor:System.ComponentModel.Component
{
    List<Worker> tlist = new List<Worker>();
    public void Start()
    {
        foreach(string url in urlList){
            // Create the thread object. This does not start the thread.
            Worker workerObject = new Worker();
            Thread workerThread = new Thread(workerObject.DoWork);

            // Start the worker thread.
            workerThread.Start(url);
            tlist.Add(workerThread);
        }
    }
}

public class Worker
{
    // This method will be called when the thread is started.
    public void DoWork(string url)
    {
        // prepare the web page we will be asking for
        HttpWebRequest  request  = (HttpWebRequest) 
            WebRequest.Create(url);

        // execute the request
        HttpWebResponse response = (HttpWebResponse)
            request.GetResponse();

        // we will read data via the response stream
        Stream resStream = response.GetResponseStream();

        // process stream
    }
}

现在我必须找到如何取消所有请求的最佳方法。

一种方法是将每个同步WebRequest转换为异步，并使用WebRequest.Abort取消处理。

另一种方法是释放线程指针并允许所有线程使用GC死掉。

Answer 1

如果要下载1000个文件，一次启动1000个线程肯定不是最佳选择。与一次只下载几个文件相比，它不仅可能不会让你获得任何加速，它还需要至少1 GB的虚拟内存。创建线程很昂贵，尽量避免在循环中这样做。

您应该做的是使用Parallel.ForEach()以及请求和响应操作的异步版本。例如像这样（WPF代码）：

private void Start_Click(object sender, RoutedEventArgs e)
{
    m_tokenSource = new CancellationTokenSource();
    var urls = …;
    Task.Factory.StartNew(() => Start(urls, m_tokenSource.Token), m_tokenSource.Token);
}

private void Cancel_Click(object sender, RoutedEventArgs e)
{
    m_tokenSource.Cancel();
}

void Start(IEnumerable<string> urlList, CancellationToken token)
{
    Parallel.ForEach(urlList, new ParallelOptions { CancellationToken = token },
                     url => DownloadOne(url, token));

}

void DownloadOne(string url, CancellationToken token)
{
    ReportStart(url);

    try
    {
        var request = WebRequest.Create(url);

        var asyncResult = request.BeginGetResponse(null, null);

        WaitHandle.WaitAny(new[] { asyncResult.AsyncWaitHandle, token.WaitHandle });

        if (token.IsCancellationRequested)
        {
            request.Abort();
            return;
        }

        var response = request.EndGetResponse(asyncResult);

        using (var stream = response.GetResponseStream())
        {
            byte[] bytes = new byte[4096];

            while (true)
            {
                asyncResult = stream.BeginRead(bytes, 0, bytes.Length, null, null);

                WaitHandle.WaitAny(new[] { asyncResult.AsyncWaitHandle,
                                           token.WaitHandle });

                if (token.IsCancellationRequested)
                    break;

                var read = stream.EndRead(asyncResult);

                if (read == 0)
                    break;

                // do something with the downloaded bytes
            }
        }

        response.Close();
    }
    finally
    {
        ReportFinish(url);
    }
}

这样，当您取消操作时，将取消所有下载并且不会启动任何新下载。此外，您可能希望设置MaxDegreeOfParallelism ParallelOptions，这样您就不会同时进行太多下载。

我不确定您要对要下载的文件做什么，因此使用StreamReader可能是更好的选择。

Answer 2

我认为最好的解决方案是“并行Foreach取消”。请检查以下代码。

要实施取消，您首先要CancellationTokenSource并通过Parallel.ForEach将其传递给option。
如果您想取消，可以致电CancellationTokenSource.Cancel()
取消后，将发生OperationCanceledException，您需要处理。

有一篇关于Parallel Programming与我的回答相关的好文章， Task Parallel Library By Sacha Barber on CodeProject 。

CancellationTokenSource tokenSource = new CancellationTokenSource();
ParallelOptions options = new ParallelOptions()
{
    CancellationToken = tokenSource.Token
};

List<string> urlList = null;
//parallel foreach cancellation
try
{
    ParallelLoopResult result = Parallel.ForEach(urlList, options, (url) =>
    {
        // Create the thread object. This does not start the thread.
        Worker workerObject = new Worker();
        workerObject.DoWork(url);
    });
}
catch (OperationCanceledException ex)
{
    Console.WriteLine("Operation Cancelled");
}

更新

以下代码为“Parallel Foreach Cancellation Sample Code”。

class Program
{
    static void Main(string[] args)
    {
        List<int> data = ParallelEnumerable.Range(1, 10000).ToList();

        CancellationTokenSource tokenSource = new CancellationTokenSource();

        Task cancelTask = Task.Factory.StartNew(() =>
            {
                Thread.Sleep(1000);
                tokenSource.Cancel();
            });


        ParallelOptions options = new ParallelOptions()
        {
            CancellationToken = tokenSource.Token
        };


        //parallel foreach cancellation
        try
        {
            Parallel.ForEach(data,options, (x, state) =>
            {
                Console.WriteLine(x);
                Thread.Sleep(100);
            });
        }
        catch (OperationCanceledException ex)
        {
            Console.WriteLine("Operation Cancelled");
        }


        Console.ReadLine();
    }
}

Answer 3

另一种方法是使用线程中止，检查Implement C# Generic Timeout并考虑生成一个你被Mark Gravel

提及的杀死的AppDomain

管理多个WebRequest的更好方法

3 个答案: