在标记为重复项之前,请先阅读问题。这个问题实现了this question中提供的解决方案,但仍然遇到僵局。
我正在调试一个大型多线程应用程序,该应用程序使用.Net客户端库对许多Google API进行了许多并发调用,有时我们在某些请求中遇到死锁。有关该应用程序的一些信息:
具体来说,我们正在调用Execute()方法,该方法使用异步方法并在等待结果时进行阻塞
public TResponse Execute()
{
try
{
using (var response = ExecuteUnparsedAsync(CancellationToken.None).Result)
{
return ParseResponse(response).Result;
}
}
...
}
这依次调用执行HttpClient.SendAsync()的ExecuteUnparsedAsync()
private async Task<HttpResponseMessage> ExecuteUnparsedAsync(CancellationToken cancellationToken)
{
using (var request = CreateRequest())
{
return await service.HttpClient.SendAsync(request, cancellationToken).ConfigureAwait(false);
}
}
现在,我了解在这里我们有很多方法可以遇到死锁,并且最好将应用程序更改为使用异步方法来避免死锁。不幸的是,这将是一笔巨大的时间投资,目前尚无法实现,但可能会在将来发生。
我的特定问题是,所有调用线程都不在线程池中,并且由于正在调用ConfigureAwait(false)
,所以我希望继续操作将始终在线程池中运行,但事实并非如此。取而代之的是,似乎在原始调用线程上计划了继续,并且线程正在死锁,因为它正在等待结果。
使用以下MCVE,我可以在几个小时内产生死锁。
using Google.Apis.Auth.OAuth2;
using Google.Apis.Drive.v2;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace DeadlockTest
{
class Program
{
const int NUM_THREADS = 70;
static long[] s_lastExecute = new long[NUM_THREADS];
static long count = 0;
static void Main(string[] args)
{
ServicePointManager.DefaultConnectionLimit = 50;
for(int i = 0; i < s_lastExecute.Length; i++)
{
s_lastExecute[i] = DateTime.Now.ToBinary();
}
Thread deadlockCheck = new Thread(new ThreadStart(CheckForDeadlock));
deadlockCheck.Start();
RunThreads();
deadlockCheck.Join();
}
static void RunThreads()
{
List<Thread> threads = new List<Thread>();
for (int i = 0; i < NUM_THREADS; i++)
{
int threadIndex = i;
Thread thread = new Thread(
new ParameterizedThreadStart(BeginThread));
thread.Start(threadIndex);
threads.Add(thread);
}
foreach(var thread in threads)
{
thread.Join();
}
}
static void BeginThread(object threadIndex)
{
Debug.Assert(SynchronizationContext.Current == null);
Debug.Assert(Thread.CurrentThread.IsThreadPoolThread == false);
ThreadLoop((int)threadIndex);
}
static void ThreadLoop(int threadIndex)
{
Random random = new Random(threadIndex);
while (true)
{
try
{
GoogleDrive.Test(random);
}
catch(Exception ex)
{
Console.WriteLine(ex.Message);
}
Interlocked.Exchange(ref s_lastExecute[threadIndex], DateTime.Now.ToBinary());
Interlocked.Increment(ref count);
}
}
private static void CheckForDeadlock()
{
Console.WriteLine("Deadlock check started");
TimeSpan period = TimeSpan.FromMinutes(1);
TimeSpan deadlockThreshold = TimeSpan.FromMinutes(10);
while (true)
{
Thread.Sleep((int)period.TotalMilliseconds);
DateTime now = DateTime.Now;
TimeSpan oldestUpdate = TimeSpan.MinValue;
for (int i = 0; i < NUM_THREADS; i++)
{
DateTime lastExecute = DateTime.FromBinary(
Interlocked.Read(ref s_lastExecute[i]));
TimeSpan delta = now - lastExecute;
if(delta > oldestUpdate)
{
oldestUpdate = delta;
}
if (delta > deadlockThreshold)
{
var msg = string.Format("Deadlock detected in thread {0} for {1} minutes",
i.ToString(), (now - lastExecute).TotalMinutes);
Console.WriteLine(msg);
}
}
int workerThreads, completionPortThreads;
System.Threading.ThreadPool.GetAvailableThreads(out workerThreads, out completionPortThreads);
Console.WriteLine("Checked for deadlocks.");
Console.WriteLine("\tWorker threads: " + workerThreads.ToString());
Console.WriteLine("\tCompletion port threads: " + completionPortThreads.ToString());
Console.WriteLine("\tExecute calls: " + Interlocked.Read(ref count).ToString());
Console.WriteLine("\tOldest update (minutes): " + oldestUpdate.TotalMinutes.ToString());
}
}
}
class GoogleDrive
{
const string SERVICE_ACCOUNT = @"<path_to_service_account>";
static string[] SCOPES = { DriveService.Scope.Drive };
public static DriveService GetDriveService(string user)
{
GoogleCredential credential;
using (var stream = new FileStream(SERVICE_ACCOUNT, FileMode.Open, FileAccess.Read))
{
credential = GoogleCredential
.FromStream(stream)
.CreateScoped(SCOPES)
.CreateWithUser(user);
}
var service = new DriveService(new DriveService.Initializer()
{
HttpClientInitializer = credential
});
return service;
}
public static void Test(Random random)
{
int userIndex = random.Next(Users.USERS.Length);
string user = Users.USERS[userIndex];
using (DriveService service = GetDriveService(user))
{
var request = service.Files.List();
var result = request.Execute();
}
}
}
public static class Users
{
public static string[] USERS = new string[]
{
"user0000@domain.com",
"user0001@domain.com",
...
};
}
}
整夜运行该测试给了我以下内容:
Deadlock detected in thread 15 for 274.216744496667 minutes
Deadlock detected in thread 45 for 154.73506413 minutes
Deadlock detected in thread 46 for 844.978023301667 minutes
Checked for deadlocks.
Worker threads: 2045
Completion port threads: 989
Execute calls: 2153228
Oldest update (minutes): 844.978023301667
一旦检测到死锁,我可以在线程循环中插入一个中断并让正在运行的线程退出。这给我留下了主线程,计时器线程,运行时使用的两个线程以及我的死锁线程(在这种情况下为三个)。还要注意,线程ID并不匹配,因为我不够聪明,无法使用实际的线程ID):
每个线程具有以下调用堆栈:
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext)
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)
mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>.GetResultCore(bool waitCompletionNotification)
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.Result.get()
Google.Apis.dll!Google.Apis.Requests.ClientServiceRequest<Google.Apis.Drive.v2.Data.FileList>.Execute()
> DeadlockTest.exe!DeadlockTest.GoogleDrive.Test(System.Random random)
DeadlockTest.exe!DeadlockTest.Program.ThreadLoop(int threadIndex)
DeadlockTest.exe!DeadlockTest.Program.BeginThread(object threadIndex)
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj)
这也让我剩下了以下任务:
我不知道确切地确定任务调度在哪个线程上的方法,但是我认为很明显,它们是为死锁线程调度的(即,它们是死锁的源)。
这使我想到了问题:
答案 0 :(得分:0)
我要假设的真正目标是消除僵局,为此,我不认为继续运行的线程甚至没有意义。我看到2个问题:
Google的Execute()方法是完全错误的。我什至会考虑报告一个错误,因为HttpClient
不支持同步调用,并且使用.Result
阻止调用是死锁期的邀请。没办法解决。
您可能通过将新线程强制加入混合来加重死锁的可能性。一个常见的误解是并发需要线程,并且在I / O绑定的情况下,这是不正确的。等待I / O结果(您的代码可能会花费大部分时间在做什么)需要no thread at all。底层子系统很复杂,但是可以通过HttpClient
之类的异步任务返回API优雅地抽象出来。使用任务代替线程,这些子系统将决定何时需要新线程,何时不需要。
所有这些都会得出您希望避免的结论-在代码中实现异步。您说这是不可行的,因此,我讨厌建议不是100%正确的代码,希望有一个公平的折衷办法是让步,以换取更大程度地减少死锁的可能性。
如果可行,我的建议是重构您的线程创建方法(MCVE中的RunThreads
)以使用Tasks(改称为RunTasks
),并将所有内容转换为 down < / em>异步调用堆栈,以对Google ExecuteAsync
而不是Execute
的调用结束。重要部分的外观如下:
static void RunTasks()
{
List<Task> tasks = new List<Task>();
for (int i = 0; i < NUM_TASKS; i++)
{
tasks.Add(BeginTaskAsync(i));
}
// Your long-term goal should be to replace this with await Task.WhenAll(tasks);
Task.WaitAll(tasks);
}
// work down your call stack here and convert methods to use async/await,
// eventually calling await ExecuteAsync() from the Google lib...
static async Task BeginTaskAsync(int taskIndex)
{
...
await ThreadLoopAsync(taskIndex);
}
static async Task ThreadLoopAsync(int taskIndex)
{
Random random = new Random(taskIndex);
while (true)
{
try
{
await GoogleDrive.TestAsync(random);
}
...
}
}
class GoogleDrive
{
...
public static async Task TestAsync(Random random)
{
...
using (DriveService service = GetDriveService(user))
{
var request = service.Files.List();
var result = await request.ExecuteAsync();
}
}
}
我提到的“妥协”是对Task.WaitAll
的调用。那是一个阻塞的调用,因此仍然不能保证您不会在此处遇到死锁。但是,如果您没有时间/资源完全使调用栈 up 一直保持异步,那么这将是一个很大的改进。您的线程阻塞要少得多,一般来说线程要少得多。