Question

我在API日志中发现了很多OperationCanceledException个。我认为它与高负载条件有关，但这可能只是因为在更高负载下，调用次数更多，因此异常数量更多。异常的堆栈跟踪未到达任何控制器代码;它通过我的一些OWIN中间件，但在它到达任何控制器之前死掉。对于许多对不同API端点的调用，它是相同的堆栈跟踪：

System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MyDomain.Api.Handlers.ApiCultureHandler.<SendAsync>d__0.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\Api\Handlers\ApiCultureHandler.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.AuditLogMiddleware.<Invoke>d__1.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\AuditLogMiddleware.cs:line 23
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.ShardingMiddleware.<Invoke>d__2.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\ShardingMiddleware.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.StructureMapMiddleware.<Invoke>d__2.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\StructureMapMiddleware.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.ExceptionHandlerMiddleware.<Invoke>d__3.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\ExceptionHandlerMiddleware.cs:line 33

这非常令人不安，因为我不确定如何调试它。

其他可能有用的信息：当这些错误发生时，我的一个API服务器在CPU使用中展示了我所谓的“Bart Simpson”模式。 CPU会在95-100％范围内射击约20-30秒，然后再下降到5-10％再持续20-30秒，然后再次恢复。这个循环无限期地重复。因为我们正在制作中，我没有时间进一步调查，我杀了那台服务器并让我们的自动调节规则调出一台新的服务器，然后工作正常。但我不能保证Bart Simpson模式不会影响我们提出的任何其他API服务器，我不知道是什么原因导致它。

有什么想法吗？

Answer 1

由于调用堆栈中的代码中没有其他错误，因此这可能是错误警报。如果客户端在服务器响应之前关闭连接，则操作将被取消。例如，假设您的客户端有一个每5秒调用一次的心跳端点，并且该调用需要250毫秒。每次客户端关闭时，都有5％的机会取消操作。在收到响应之前，有许多正常情况下客户端可能会切断连接。 OperationCanceledException本身不应该是警报的原因。但是，请考虑与客户数量和特定应用相关的频率。

Answer 2

正如所讨论的，有可能在某个地方，在堆栈跟踪中列出的其中一个自定义中间件中，可能混合了 async await 和阻塞调用（.Result或.Wait()）可能导致死锁导致操作挂起，超时和取消。

堆栈跟踪片段已经提供了可能的嫌疑人列表。

建议首先检查处理请求管道的列出的中间件的Invoke。

在OWIN管道中导致OperationCanceledException的原因是什么？

2 个答案: