Question

我想下载一个包含6个线程的文件来加速这个过程，所以这是计算每个线程文件大小的部分代码：

string url = "http://somefile.mp3";
List<FileDownloader> filewonloadersList = new List<FileDownloader>();
System.Net.WebRequest req = System.Net.HttpWebRequest.Create(url);
var response = req.GetResponse();
req.Method = "HEAD";
System.Net.WebResponse resp = req.GetResponse();
int responseLength = int.Parse(resp.Headers.Get("Content-Length"));
int parts = 6;
var eachSize = responseLength / parts;
var lastPartSize = eachSize + (responseLength % parts);
for (int i = 0; i < parts - 1; i++)
{
    filewonloadersList.Add(new FileDownloader(url, i * eachSize, eachSize));
}
filewonloadersList.Add(new FileDownloader(url, (parts - 1) * eachSize, lastPartSize));
var threads = new List<Thread>();
foreach (var item in filewonloadersList)
{
    var newThread = new Thread(DoDownload);
    threads.Add(newThread);
    newThread.Start(item);
}

这是DoDownload函数的主体：

public static void DoDownload(object data)
{
    retry:
        try
        {
            var downloader = data as FileDownloader;
            HttpWebRequest req = (HttpWebRequest)WebRequest.Create(downloader.Url);
            if (downloader.Start > 0)
            {
                req.AddRange(downloader.Start, downloader.Start + downloader.Count - 1);
            }
            else
            {
                req.AddRange(downloader.Start, downloader.Start + downloader.Count - 1);
            }
            var response = req.GetResponse();
            using (var reponseStream = response.GetResponseStream())
            {
                using (var fs = new FileStream($"temp_{downloader.Start}.sth", FileMode.OpenOrCreate))
                {
                    var buffer = new byte[1024];
                    int bytesRead = 0;
                    do
                    {
                        bytesRead = reponseStream.Read(buffer, 0, 1024);
                        fs.Write(buffer, 0, bytesRead);
                        fs.Flush();
                    } while (bytesRead > 0);
                    fs.Close();
                }
            }
        }
        catch (WebException e)
        {
            if (e.Status == WebExceptionStatus.Timeout || e.Status == WebExceptionStatus.KeepAliveFailure)
                goto retry;
        }
    }

此外，这是FileDownloader定义：

public class FileDownloader
{
    public int Start;
    public int Count;
    public string PathTemp;
    public string Url;
    public FileDownloader(string url, int start, int count)
    {
        Url = url;
        Start = start;
        Count = count;
    }
}

一切都按照我的预期工作，文件的长度与应有的一样多。在我合并下载文件的部分之后，它也正常工作。问题是线程部分。我预计会同时下载6个文件，但它们是逐个下载的，例如第一部分完成后，将下载第二部分。我该如何解决这个问题？

更新

根据建议，我已将功能更改为async：

public async Task DoDownload(object data)
        {
            retry:
            try
            {
                var downloader = data as FileDownloader;
                HttpWebRequest req = (HttpWebRequest)WebRequest.Create(downloader.Url);
                req.AddRange(downloader.Start, downloader.Start + downloader.Count - 1);
                var response = await req.GetResponseAsync();
                using (var reponseStream = response.GetResponseStream())
                {
                    using (var fs = new FileStream($"temp_{downloader.Start}.sth", FileMode.OpenOrCreate))
                    {
                        var buffer = new byte[1024];
                        int bytesRead = 0;
                        do
                        {
                            //reponseStream.Seek(downloader.Start, SeekOrigin.Current);
                            bytesRead = await reponseStream.ReadAsync(buffer, 0, 1024);
                            await fs.WriteAsync(buffer, 0, bytesRead);
                            await fs.FlushAsync();
                        } while (bytesRead > 0);
                        fs.Close();
                    }
                }
            }
            catch (WebException e)
            {
                if (e.Status == WebExceptionStatus.Timeout || e.Status == WebExceptionStatus.KeepAliveFailure)
                    goto retry;
            }
        }

部分填充的foreach循环：

foreach (var item in filewonloadersList)
            {
                Task.WhenAll(DoDownload(item));
            }

但结果是一样的！只有一个文件的一部分同时下载。

Answer 1

基于此链接Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry)。简而言之，这是ServicePoint的问题。它提供HTTP连接的连接管理。 ServicePoint对象允许的默认最大并发连接数为2.

我只需要在代码中添加一个不错的行：

  COL_A COL_B COL_C                 COL_D   COL_Y
0  VAL1  VAL2  VAL3  OFFER1|OFFER2|OFFER3  OFFER1
1  VAL1  VAL2  VAL3  OFFER1|OFFER2|OFFER3  OFFER2
2  VAL1  VAL2  VAL3  OFFER1|OFFER2|OFFER3  OFFER3

它的工作就像一个魅力！同时有多个文件，并且异步和线程的工作方式也没有区别。至少他们两个都在产生我想要的结果。

通过多个线程下载文件

1 个答案: