Question

该应用程序解析某些目录中的文件，同时将新文件添加到目录中。我使用ConcurrentQueue并尝试将工作分成核心数。因此，如果有要处理的文件 - 它应该同时处理多达4个（核心）文件。然而，在处理10-30个文件后，应用程序在几秒钟内运行OOM。我看到内存消耗快速增长到~1.5GB，比出现OOM错误。我是任务调度员，所以我可能做错了什么。通过在文件上运行一些.exe来完成文件解析，该文件使用＆lt; 5mb或ram。每当计时器线程过去时，任务调度程序就会运行。但它甚至在计时器第二次过去之前运行OOM。

private void OnTimedEvent(object source, ElapsedEventArgs e)
    {
        DirectoryInfo info = new DirectoryInfo(AssemblyDirectory);
        FileInfo[] allSrcFiles = info.GetFiles("*.dat").OrderBy(p => p.CreationTime).ToArray();
        var validSrcFiles = allSrcFiles.Where(p => (DateTime.Now - p.CreationTime) > TimeSpan.FromSeconds(60));
        var newFilesToParse = validSrcFiles.Where(f => !ProcessedFiles.Contains(f.Name));
        if (newFilesToParse.Any()) Console.WriteLine("Adding " + newFilesToParse.Count() + " files to the Queue");
        foreach (var file in newFilesToParse)
        {
            FilesToParseQueue.Enqueue(file);
            ProcessedFiles.Add(file.Name);
        }
        if (!busy)
        {

            if (FilesToParseQueue.Any())
            {
                busy = true;
                Console.WriteLine("");
                Console.WriteLine("There are " + FilesToParseQueue.Count + " files in queue. Processing...");
            }
            var scheduler = new LimitedConcurrencyLevelTaskScheduler(coresCount); //4
            TaskFactory factory = new TaskFactory(scheduler);
            while (FilesToParseQueue.Any())
            {
                factory.StartNew(() =>
                {
                    FileInfo file;
                    if (FilesToParseQueue.TryDequeue(out file))
                    {
                        //Dequeue();
                        ParseFile(file);
                    }
                });
            }
            if (!FilesToParseQueue.Any())
            {
                busy = false;
                Console.WriteLine("Finished processing Files in the Queue. Waiting for new files...");
            }
        }
    }

Answer 1

只要有要处理的文件，您的代码就会继续创建新的Task，并且它可以更快地处理文件。但它没有其他限制（如目录中的文件数），这就是它快速耗尽内存的原因。

一个简单的解决方法是在循环外移动出队：

while (true)
{
    FileInfo file;
    if (FilesToParseQueue.TryDequeue(out file))
    {
        factory.StartNew(() => ParseFile(file));
    }
    else
    {
        break;
    }
}

如果每个核心只创建一个Task并使用Task内的循环处理文件，那么性能会更好。

Answer 2

这种问题（你排队多个工作单元，并希望它们在 parallel 中处理）非常适合TPL Dataflow：

private async void OnTimedEvent(object source, ElapsedEventArgs e)
{
  DirectoryInfo info = new DirectoryInfo(AssemblyDirectory);
  FileInfo[] allSrcFiles = info.GetFiles("*.dat").OrderBy(p => p.CreationTime).ToArray();
  var validSrcFiles = allSrcFiles.Where(p => (DateTime.Now - p.CreationTime) > TimeSpan.FromSeconds(60));
  var newFilesToParse = validSrcFiles.Where(f => !ProcessedFiles.Contains(f.Name));
  if (newFilesToParse.Any()) Console.WriteLine("Adding " + newFilesToParse.Count() + " files to the Queue");

  var blockOptions = new ExecutionDataflowBlockOptions
  {
    MaxDegreeOfParallelism = coresCount,
  };
  var block = new ActionBlock<FileInfo>(ParseFile, blockOptions);

  var filesToParseCount = 0;
  foreach (var file in newFilesToParse)
  {
    block.Post(file);
    ProcessedFiles.Add(file.Name);
    ++filesToParseCount;
  }
  Console.WriteLine("There are " + filesToParseCount + " files in queue. Processing...");

  block.Complete();
  await block.Completion;
  Console.WriteLine("Finished processing Files in the Queue. Waiting for new files...");
}

Answer 3

基本解决方案

您可以通过将代码拆分为如下所示的基本要素来实际修复代码：

// This is technically a misnomer. It should be
// called "FileNamesQueuedForProcessing" or similar.
// Non-thread-safe. Assuming timer callback access only.
private readonly HashSet<string> ProcessedFiles = new HashSet<string>();
private readonly LimitedConcurrencyLevelTaskScheduler LimitedConcurrencyScheduler = new LimitedConcurrencyLevelTaskScheduler(Environment.ProcessorCount);

private void OnTimedEvent(object source, ElapsedEventArgs e)
{
    DirectoryInfo info = new DirectoryInfo(AssemblyDirectory);

    // Slightly rewritten to cut down on allocations.
    FileInfo[] newFilesToParse = info
        .GetFiles("*.dat")
        .Where(f =>
            (DateTime.Now - f.CreationTime) > TimeSpan.FromSeconds(60) && // I'd consider removing this filter.
            !ProcessedFiles.Contains(f.Name))
        .OrderBy(p => p.CreationTime)
        .ToArray();

    if (newFilesToParse.Length != 0) Console.WriteLine("Adding " + newFilesToParse.Count() + " files to the Queue");

    foreach (FileInfo file in newFilesToParse)
    {
        // Fire and forget.
        // You can add the resulting task to a shared thread-safe collection
        // if you want to observe completion/exceptions/cancellations.
        Task.Factory.StartNew(
            () => ParseFile(file)
            , CancellationToken.None
            , TaskCreationOptions.DenyChildAttach
            , LimitedConcurrencyScheduler
        );

        ProcessedFiles.Add(file.Name);
    }
}

请注意我自己没有进行任何类型的负载平衡，而是依赖LimitedConcurrencyLevelTaskScheduler执行宣传 - 也就是说，立即接受Task.Factory.StartNew上的所有工作项，在内部对其进行排队处理它们在未来的某个时间最多[N =最大并行度]线程池线程。

P.S。我假设OnTimedEvent将始终在同一个线程上触发。如果没有，则需要进行一些小改动以确保线程安全：

private void OnTimedEvent(object source, ElapsedEventArgs e)
{
    lock (ProcessedFiles)
    {
        // As above.
    }
}

替代解决方案

现在，这里有一个更新颖的方法：我们如何摆脱计时器和LimitedConcurrencyLevelTaskScheduler并在单个模块化管道中封装处理的所有？将会有一个很多的阻止代码（除非你打破TPL数据流 - 但我会坚持使用基类库类型），但阶段之间的消息传递是所以它很容易实现真正吸引人的设计（当然我认为）。

private async Task PipelineAsync()
{
    const int MAX_FILES_TO_BE_QUEUED = 16;

    using (BlockingCollection<FileInfo> queue = new BlockingCollection<FileInfo>(boundedCapacity: MAX_FILES_TO_BE_QUEUED))
    {
        Task producer = Task.Run(async () =>
        {
            try
            {
                while (true)
                {
                    DirectoryInfo info = new DirectoryInfo(AssemblyDirectory);
                    HashSet<string> namesOfFilesQeueuedForProcessing = new HashSet<string>();

                    FileInfo[] newFilesToParse = info
                        .GetFiles("*.dat")
                        .Where(f =>
                            (DateTime.Now - f.CreationTime) > TimeSpan.FromSeconds(60) &&
                            !ProcessedFiles.Contains(f.Name))
                        .OrderBy(p => p.CreationTime) // Processing order is not guaranteed.
                        .ToArray();

                    foreach (FileInfo file in newFilesToParse)
                    {
                        // This will block if we reach bounded capacity thereby throttling
                        // the producer (meaning we'll never overflow the handover collection).
                        queue.Add(file);
                        namesOfFilesQeueuedForProcessing.Add(file.Name);
                    }

                    await Task.Delay(TimeSpan.FromSeconds(60)).ConfigureAwait(false);
                }
            }
            finally
            {
                // Exception? Cancellation? We'll let the
                // consumer know that it can wind down.
                queue.CompleteAdding();
            }
        });

        Task consumer = Task.Run(() =>
        {
            ParallelOptions options = new ParallelOptions {
                MaxDegreeOfParallelism = Environment.ProcessorCount
            };

            Parallel.ForEach(queue.GetConsumingEnumerable(), options, file => ParseFile(file)); 
        });

        await Task.WhenAll(producer, consumer).ConfigureAwait(false);
    }
}

这种模式的一般形式在Stephen Toub的“并行编程模式”（第55页）中有所描述。我强烈推荐一下。

此处的权衡是由于使用BlockingCollection<T>和Parallel.ForEach而您将要执行的阻止数量。管道作为一个概念的好处很多：新阶段（Task实例）易于添加，完成和取消易于连接，生产者和消费者例外都被观察到，所有可变状态都令人愉快本地

使用任务计划程序的应用程序快速耗尽内存

3 个答案: