为什么我的异步继续安排在线程池之外?

时间:2018-08-23 16:06:54

标签: c# task deadlock dotnet-httpclient taskscheduler

在标记为重复项之前,请先阅读问题。这个问题实现了this question中提供的解决方案,但仍然遇到僵局。

我正在调试一个大型多线程应用程序,该应用程序使用.Net客户端库对许多Google API进行了许多并发调用,有时我们在某些请求中遇到死锁。有关该应用程序的一些信息:

  • 应用程序是Windows服务(SynchronizationContext为空)
  • .Net Framework版本为4.5
  • 所有发出API请求的线程都不在默认线程池中。

具体来说,我们正在调用Execute()方法,该方法使用异步方法并在等待结果时进行阻塞

public TResponse Execute()
{
    try
    {
        using (var response = ExecuteUnparsedAsync(CancellationToken.None).Result)
        {
            return ParseResponse(response).Result;
        }
    }
    ...
}

这依次调用执行HttpClient.SendAsync()的ExecuteUnparsedAsync()

private async Task<HttpResponseMessage> ExecuteUnparsedAsync(CancellationToken cancellationToken)
{
    using (var request = CreateRequest())
    {
        return await service.HttpClient.SendAsync(request, cancellationToken).ConfigureAwait(false);
    }
}

现在,我了解在这里我们有很多方法可以遇到死锁,并且最好将应用程序更改为使用异步方法来避免死锁。不幸的是,这将是一笔巨大的时间投资,目前尚无法实现,但可能会在将来发生。

我的特定问题是,所有调用线程都不在线程池中,并且由于正在调用ConfigureAwait(false),所以我希望继续操作将始终在线程池中运行,但事实并非如此。取而代之的是,似乎在原始调用线程上计划了继续,并且线程正在死锁,因为它正在等待结果。

使用以下MCVE,我可以在几个小时内产生死锁。

using Google.Apis.Auth.OAuth2;
using Google.Apis.Drive.v2;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace DeadlockTest
{
    class Program
    {
        const int NUM_THREADS = 70;

        static long[] s_lastExecute = new long[NUM_THREADS];
        static long count = 0;

        static void Main(string[] args)
        {
            ServicePointManager.DefaultConnectionLimit = 50;

            for(int i = 0; i < s_lastExecute.Length; i++)
            {
                s_lastExecute[i] = DateTime.Now.ToBinary();
            }

            Thread deadlockCheck = new Thread(new ThreadStart(CheckForDeadlock));
            deadlockCheck.Start();

            RunThreads();

            deadlockCheck.Join();
        }

        static void RunThreads()
        {
            List<Thread> threads = new List<Thread>();

            for (int i = 0; i < NUM_THREADS; i++)
            {
                int threadIndex = i;
                Thread thread = new Thread(
                    new ParameterizedThreadStart(BeginThread));
                thread.Start(threadIndex);
                threads.Add(thread);
            }

            foreach(var thread in threads)
            {
                thread.Join();
            }
        }

        static void BeginThread(object threadIndex)
        {
            Debug.Assert(SynchronizationContext.Current == null);
            Debug.Assert(Thread.CurrentThread.IsThreadPoolThread == false);
            ThreadLoop((int)threadIndex);
        }

        static void ThreadLoop(int threadIndex)
        {
            Random random = new Random(threadIndex);

            while (true)
            {
                try
                {
                    GoogleDrive.Test(random);
                }
                catch(Exception ex)
                {
                    Console.WriteLine(ex.Message);
                }

                Interlocked.Exchange(ref s_lastExecute[threadIndex], DateTime.Now.ToBinary());
                Interlocked.Increment(ref count);
            }
        }

        private static void CheckForDeadlock()
        {
            Console.WriteLine("Deadlock check started");

            TimeSpan period = TimeSpan.FromMinutes(1);
            TimeSpan deadlockThreshold = TimeSpan.FromMinutes(10);

            while (true)
            {
                Thread.Sleep((int)period.TotalMilliseconds);

                DateTime now = DateTime.Now;
                TimeSpan oldestUpdate = TimeSpan.MinValue;

                for (int i = 0; i < NUM_THREADS; i++)
                {
                    DateTime lastExecute = DateTime.FromBinary(
                        Interlocked.Read(ref s_lastExecute[i]));

                    TimeSpan delta = now - lastExecute;
                    if(delta > oldestUpdate)
                    {
                        oldestUpdate = delta;
                    }

                    if (delta > deadlockThreshold)
                    {
                        var msg = string.Format("Deadlock detected in thread {0} for {1} minutes",
                            i.ToString(), (now - lastExecute).TotalMinutes);
                        Console.WriteLine(msg);
                    }
                }

                int workerThreads, completionPortThreads;
                System.Threading.ThreadPool.GetAvailableThreads(out workerThreads, out completionPortThreads);

                Console.WriteLine("Checked for deadlocks.");
                Console.WriteLine("\tWorker threads: " + workerThreads.ToString());
                Console.WriteLine("\tCompletion port threads: " + completionPortThreads.ToString());
                Console.WriteLine("\tExecute calls: " + Interlocked.Read(ref count).ToString());
                Console.WriteLine("\tOldest update (minutes): " + oldestUpdate.TotalMinutes.ToString());
            }
        }
    }

    class GoogleDrive
    {
        const string SERVICE_ACCOUNT = @"<path_to_service_account>";

        static string[] SCOPES = { DriveService.Scope.Drive };

        public static DriveService GetDriveService(string user)
        {
            GoogleCredential credential;
            using (var stream = new FileStream(SERVICE_ACCOUNT, FileMode.Open, FileAccess.Read))
            {
                credential = GoogleCredential
                    .FromStream(stream)
                    .CreateScoped(SCOPES)
                    .CreateWithUser(user);
            }

            var service = new DriveService(new DriveService.Initializer()
            {
                HttpClientInitializer = credential
            });

            return service;
        }

        public static void Test(Random random)
        {
            int userIndex = random.Next(Users.USERS.Length);
            string user = Users.USERS[userIndex];
            using (DriveService service = GetDriveService(user))
            {
                var request = service.Files.List();
                var result = request.Execute();
            }
        }
    }

    public static class Users
    {
        public static string[] USERS = new string[]
        {
            "user0000@domain.com",
            "user0001@domain.com",
            ...
        };
    }
}

整夜运行该测试给了我以下内容:

Deadlock detected in thread 15 for 274.216744496667 minutes
Deadlock detected in thread 45 for 154.73506413 minutes
Deadlock detected in thread 46 for 844.978023301667 minutes
Checked for deadlocks.
        Worker threads: 2045
        Completion port threads: 989
        Execute calls: 2153228
        Oldest update (minutes): 844.978023301667

一旦检测到死锁,我可以在线程循环中插入一个中断并让正在运行的线程退出。这给我留下了主线程,计时器线程,运行时使用的两个线程以及我的死锁线程(在这种情况下为三个)。还要注意,线程ID并不匹配,因为我不够聪明,无法使用实际的线程ID):

Process threads

每个线程具有以下调用堆栈:

    mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext)
    mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)
    mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
    mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
    mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
    mscorlib.dll!System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>.GetResultCore(bool waitCompletionNotification)
    mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.Result.get()
    Google.Apis.dll!Google.Apis.Requests.ClientServiceRequest<Google.Apis.Drive.v2.Data.FileList>.Execute()
>   DeadlockTest.exe!DeadlockTest.GoogleDrive.Test(System.Random random)
    DeadlockTest.exe!DeadlockTest.Program.ThreadLoop(int threadIndex)
    DeadlockTest.exe!DeadlockTest.Program.BeginThread(object threadIndex)
    mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state)
    mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
    mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj)

这也让我剩下了以下任务:

Scheduled Tasks

我不知道确切地确定任务调度在哪个线程上的方法,但是我认为很明显,它们是为死锁线程调度的(即,它们是死锁的源)。

这使我想到了问题:

  • 为什么未在线程池而不是调用线程中安排继续任务?
  • 是否有一种方法可以强制任务在线程池上运行,但结果上没有线程池线程块?

1 个答案:

答案 0 :(得分:0)

我要假设的真正目标是消除僵局,为此,我不认为继续运行的线程甚至没有意义。我看到2个问题:

  1. Google的Execute()方法是完全错误的。我什至会考虑报告一个错误,因为HttpClient不支持同步调用,并且使用.Result阻止调用是死锁期的邀请。没办法解决。

  2. 您可能通过将新线程强制加入混合来加重死锁的可能性。一个常见的误解是并发需要线程,并且在I / O绑定的情况下,这是不正确的。等待I / O结果(您的代码可能会花费大部分时间在做什么)需要no thread at all。底层子系统很复杂,但是可以通过HttpClient之类的异步任务返回API优雅地抽象出来。使用任务代替线程,这些子系统将决定何时需要新线程,何时不需要。

所有这些都会得出您希望避免的结论-在代码中实现异步。您说这是不可行的,因此,我讨厌建议不是100%正确的代码,希望有一个公平的折衷办法是让步,以换取更大程度地减少死锁的可能性。

如果可行,我的建议是重构您的线程创建方法(MCVE中的RunThreads)以使用Tasks(改称为RunTasks),并将所有内容转换为 down < / em>异步调用堆栈,以对Google ExecuteAsync而不是Execute的调用结束。重要部分的外观如下:

static void RunTasks()
{
    List<Task> tasks = new List<Task>();

    for (int i = 0; i < NUM_TASKS; i++)
    {
        tasks.Add(BeginTaskAsync(i));
    }

    // Your long-term goal should be to replace this with await Task.WhenAll(tasks);
    Task.WaitAll(tasks);
}

// work down your call stack here and convert methods to use async/await,
// eventually calling await ExecuteAsync() from the Google lib...

static async Task BeginTaskAsync(int taskIndex)
{
    ...
    await ThreadLoopAsync(taskIndex);
}

static async Task ThreadLoopAsync(int taskIndex)
{
    Random random = new Random(taskIndex);

    while (true)
    {
        try
        {
            await GoogleDrive.TestAsync(random);
        }
        ...
    }
}

class GoogleDrive
{
    ...

    public static async Task TestAsync(Random random)
    {
        ...
        using (DriveService service = GetDriveService(user))
        {
            var request = service.Files.List();
            var result = await request.ExecuteAsync();
        }
    }
}

我提到的“妥协”是对Task.WaitAll的调用。那是一个阻塞的调用,因此仍然不能保证您不会在此处遇到死锁。但是,如果您没有时间/资源完全使调用栈 up 一直保持异步,那么这将是一个很大的改进。您的线程阻塞要少得多,一般来说线程要少得多。