Question

我正在下载100K +文件，并希望在补丁中执行此操作，例如一次100个文件。

static void Main(string[] args) {
    Task.WaitAll(
      new Task[]{
           RunAsync()
    });
}

// each group has 100 attachments.
static async Task RunAsync() {
    foreach (var group in groups) {
        var tasks = new List<Task>();
        foreach (var attachment in group.attachments) {
            tasks.Add(DownloadFileAsync(attachment, downloadPath));
        }
        await Task.WhenAll(tasks);
    }
}

static async Task DownloadFileAsync(Attachment attachment, string path) {
    using (var client = new HttpClient()) {
        using (var fileStream = File.Create(path + attachment.FileName)) {
            var downloadedFileStream = await client.GetStreamAsync(attachment.url);
            await downloadedFileStream.CopyToAsync(fileStream);
        }
    }
}

预期希望一次下载100个文件，然后下载100个文件;

实际它同时下载了很多。很快收到错误Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host

Answer 1

在＆＃34;批处理＆＃34;中运行任务在性能方面不是一个好主意。长时间运行的任务将使整个批处理块。一个完成后，更好的方法是开始一项新任务。

这可以通过@MertAkcakaya建议的队列来实现。但我会根据我的另一个答案Have a set of Tasks with only X running at a time

发布另一个备选方案

int maxTread = 3;
System.Net.ServicePointManager.DefaultConnectionLimit = 50; //Set this once to a max value in your app

var urls = new Tuple<string, string>[] {
    Tuple.Create("http://cnn.com","temp/cnn1.htm"),
    Tuple.Create("http://cnn.com","temp/cnn2.htm"),
    Tuple.Create("http://bbc.com","temp/bbc1.htm"),
    Tuple.Create("http://bbc.com","temp/bbc2.htm"),
    Tuple.Create("http://stackoverflow.com","temp/stackoverflow.htm"),
    Tuple.Create("http://google.com","temp/google1.htm"),
    Tuple.Create("http://google.com","temp/google2.htm"),
};
DownloadParallel(urls, maxTread);

async Task DownloadParallel(IEnumerable<Tuple<string,string>> urls, int maxThreads)
{
    SemaphoreSlim maxThread = new SemaphoreSlim(maxThreads);
    var client = new HttpClient();

    foreach(var url in urls)
    {
        await maxThread.WaitAsync();
        DownloadFile(client, url.Item1, url.Item2)
                    .ContinueWith((task) => maxThread.Release() );
    }
}


async Task DownloadFile(HttpClient client, string url, string fileName)
{
    var stream = await client.GetStreamAsync(url);
    using (var fileStream = File.Create(fileName))
    {
        await stream.CopyToAsync(fileStream);
    }
}

PS： DownloadParallel 将在上次开始下载后立即返回。所以不要等待它。如果你真的想要等待它，你应该在方法的末尾添加for (int i = 0; i < maxThreads; i++) await maxThread.WaitAsync();。

PS2：别忘了将异常处理添加到 DownloadFile

在for循环中每隔x次运行Async

1 个答案: