Azure WebJob日志记录崩溃主机,我们可以做什么?

时间:2016-04-04 09:29:58

标签: azure azure-webjobs

我们使用Azure运行多个WebJobs。我们的一个WebJob函数具有以下签名:

public static Task ProcessFileUploadedMessageAsync(
    [QueueTrigger("uploads")] FileUploadedMessage message,
    TextWriter logger,
    CancellationToken cancellationToken)

此函数正在监视一个消息队列,该消息指示已上载文件,然后触发文件数据的导入。请注意使用TextWriter作为第二个参数:这是由WebJobs API基础结构提供的。

我们的导入过程有点慢(在某些情况下单个文件导入可能需要几个小时),因此我们会定期将消息写入日志(通过TextWriter)以跟踪我们的进度。不幸的是,我们较大的文件导致WebJob托管进程因日志记录异常而终止。以下是托管进程日志中的示例堆栈跟踪:

[04/02/2016 03:44:59 > 660083: ERR ] 
[04/02/2016 03:45:00 > 660083: ERR ] Unhandled Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
[04/02/2016 03:45:00 > 660083: ERR ] Parameter name: chunkLength
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Text.StringBuilder.ToString()
[04/02/2016 03:45:00 > 660083: ERR ]    at System.IO.StringWriter.ToString()
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<TryExecuteAsync>d__3.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.RecurrentTaskSeriesCommand.<ExecuteAsync>d__0.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
[04/02/2016 03:45:00 > 660083: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/02/2016 03:45:00 > 660083: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[04/02/2016 03:45:00 > 660083: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[04/02/2016 03:45:00 > 660083: SYS ERR ] Job failed due to exit code -532462766
[04/02/2016 03:45:01 > 660083: SYS INFO] Process went down, waiting for 0 seconds
[04/02/2016 03:45:01 > 660083: SYS INFO] Status changed to PendingRestart

主要问题是这个异常不是由我们的代码抛出,而是由WebJobs API中的某些东西引发:

at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()

我们尝试将try...catch块放在我们对TextWriter的调用周围,但这些块无效。看起来日志消息在某处缓冲,并通过单独的线程定期刷新到Azure blob存储。如果是这种情况,那么我们就无法捕获异常。

是否还有其他人遇到同样的问题,或者有人会想到可能的解决方案或解决方法吗?

为了完整起见,我们使用TextWriter进行日志记录的方式如下:

await logger.WriteLineAsync("some text to log");

没有比那更复杂的了。

更新:好像已报告为issue #675 on the WebJobs SDK GitHub

1 个答案:

答案 0 :(得分:0)

您应该向方法添加async,就像这样:

public async static Task ProcessFileUploadedMessageAsync(...)

要获取详细信息,您可以看到this article