NodeJS Lambda似乎失败,但报告成功

时间:2019-10-17 03:48:59

标签: node.js aws-lambda

我看到了我不完全理解的NodeJS Lambda上的行为。首先,我假设我们的代码中至少存在一个错误,但是我们难以理解的问题是,它如何影响同一Lambda实例上的子序列调用。

尽我所能理解的问题是,在初始失败之后,我的下一次调用也会失败,并且我看到一个错误,似乎表明nodejs已终止。然后我看到它重新初始化,并再次成功开始处理请求。

有一些问题:

  1. 好的,我们似乎有一个错误,我想我们可以对其进行跟踪并在某个时间点对其进行修复
  2. 随后的调用中的nodejs终止,但调用被视为成功(当然不会成功)
  3. 因为第二次调用被视为成功,所以不会重试

让我尝试展示一下外观的一些细节。首先,这是第二次调用,实际上确实失败了,NodeJS似乎终止了:

START RequestId: a71447b0-19f8-456c-b163-47e0408bf3a1 Version: $LATEST
2019-10-17T00:26:12.295Z    a71447b0-19f8-456c-b163-47e0408bf3a1    INFO    Assuming lambda role: { ... }lambda',
END RequestId: a71447b0-19f8-456c-b163-47e0408bf3a1
REPORT RequestId: a71447b0-19f8-456c-b163-47e0408bf3a1    Duration: 228.19 ms    Billed Duration: 300 ms    Memory Size: 256 MBMax Memory Used: 158 MB
RequestId: a71447b0-19f8-456c-b163-47e0408bf3a1 Error: Runtime exited with error: exit status 129

也许我误解了最后一行,但是肯定看起来NodeJS对我终止了(使用SIGHUP ??!)。之后,下一次调用将显示XRay日志记录,就像在该实例的第一次调用上看到的一样:

2019-10-17T00:55:30.531Z       undefined       INFO    AWS_XRAY_DAEMON_ADDRESS is set. Configured daemon address to 169.xxx.xxx.xxx.
2019-10-17T00:55:30.536Z       undefined       INFO    AWS_XRAY_CONTEXT_MISSING is set. Configured context missing strategy to LOG_ERROR.
2019-10-17T00:55:30.554Z       undefined       INFO    Subsegment streaming threshold set to: 0

所以它在我的lambda实例上重新启动了NodeJ?那是我们应该看到的吗?

如果我通过X射线等进行查找,则显示错误,但状态为200:

Screenshot from AWS X-Ray

我认为值得一提的是,先前的调用似乎触发了这一点。它也成功,并且确实返回了正确的值,但完成后,我在日志中看到了这一点:

2019-10-17T00:55:29.169Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.326Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.328Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.329Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.329Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.330Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.368Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.388Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.408Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.428Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.467Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.468Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.469Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.567Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}
2019-10-17T00:55:29.568Z    3bf23bda-e7fa-4fd7-b373-2258baff2bb8    ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"premature close","stack":["Error: premature close","    at Duplexify.onclose (/var/task/node_modules/end-of-stream/index.js:47:67)","    at Duplexify.emit (events.js:198:13)","    at Duplexify.EventEmitter.emit (domain.js:448:20)","    at Duplexify._destroy (/var/task/node_modules/duplexify/index.js:199:8)","    at /var/task/node_modules/duplexify/index.js:182:10","    at /var/task/node_modules/async-listener/glue.js:188:31","    at process._tickCallback (internal/process/next_tick.js:61:11)"]}

毫无疑问,这是我们代码中的错误。对我来说,这可能不是很明显,因为这些库不是我们的代码使用的库。实际上,这些似乎已被AWS XRay使用。那么XRay是否存在故障,并且导致我们调用失败?

我想了解:

  • nodejs是否失败并正在重新启动?
  • 是否有办法捕获此过程终止并将其至少视为失败的调用?
  • 简而言之:这是怎么回事?它似乎与我们通常期望在lambda中看到的结果不符...

以下是一些我们要尝试的事情,我们尝试了但没有帮助:

  • 升级了aws-sdk和xray软件包:没有变化
  • 我们已经添加了异常处理和日志记录功能,而没有学习任何新知识,我们将继续沿这条路前进

更新:在“正常”情况下,包括记录了多个错误的情况,我们将做同样的事情。我们订阅了一个IoT主题,发布了一条消息,并等待长达5秒钟的响应。这些执行都没有等待响应的超时(我们最多等待5秒钟,如果未收到响应则返回错误),并且Lambda执行都没有超时。在上述情况下(请求ID a71447b0-19f8-456c-b163-47e0408bf3a1在日志中),我们不这样做,因为据我所知我们的代码未运行-没有日志表明它可以运行。

这种情况不会经常发生。在过去几天中,我正在执行25次执行中的大约1次。负担不大,因此我们只执行了大约190次调用(到目前为止,其中8次失败)。

1 个答案:

答案 0 :(得分:0)

几个月后返回报告,以描述这是什么以及如何解决。

尚不清楚(但事后看来,应该是这样),其根源是专门在lambda中使用MQTT.js(作为AWS IOT SDK的一部分)。特别是当lambda不活动时,有时会断开连接-这是可以预期的。这里发生的是MQTT.js在断开连接时触发一个事件,并且该事件可以在不可预知的时间运行。

总而言之,我们需要处理errorclose事件。如果未处理它们,则如上所述,我们将遇到未处理的异常。解决这个问题非常简单-我们只需要处理这些事件并创建一个新的客户端等。

我认为我们应该比以前更简单地解决此问题-对我们而言,它与MQTT无关(但一旦我们意识到这是相对较快解决的,对我们而言并不明显)。


作为旁注,可能值得指出的是,最新的IoT专用AWS开发工具包无论如何都不使用MQTTjs。我不确切知道这样做的动机是什么,但是我不知道这个问题/答案在未来的改变中会有多大用处。