无法在Apache气流中执行异步操作

时间:2019-07-03 10:28:19

标签: node.js asynchronous airflow

从气流运行node.js命令时,我无法执行任何异步操作。我正在使用bash运算符来触发节点脚本。脚本中的所有同步操作都可以正常工作,但是一旦执行了一些异步操作,它就会异常退出。

我正在使用Apache Airflow 1.10.3执行数据管道。我需要触发具有异步I / O的node.js文件。 BashOperator可以很好地运行所有先前的作业。一旦我触发了节点代码,它将执行代码中的任何同步部分,但是当出现一些异步函数时,代码将终止。我尝试了一个简单的代码,在控制台上打印了一些值,然后进行了设置超时并在超时内进行了控制台。如果我在普通的CLI上运行此代码,它将运行良好,但是当在气流中使用BashOperator运行相同的代码时,它将在打印第一个同步控制台后终止。我尝试从shell脚本,scala编译代码,python代码和C代码调用node.js文件,并在气流上运行时遇到相同的错误。

console.log("This console is working");

setTimeout(function(){console.log("This console is working after 
timeout");},5000);

以下是我在运行上述代码时收到的错误消息

`[2019-07-03 12:36:32,803] {bash_operator.py:81} INFO - Tmp dir root 
location: 
 /tmp
[2019-07-03 12:36:32,803] {bash_operator.py:90} INFO - Exporting the 
following env vars:
AIRFLOW_CTX_DAG_ID=Node_Check_Anokhi
AIRFLOW_CTX_TASK_ID=Node_Test
AIRFLOW_CTX_EXECUTION_DATE=2019-07-03T07:06:00.488520+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2019-07-03T07:06:00.488520+00:00
[2019-07-03 12:36:32,803] {bash_operator.py:104} INFO - Temporary script 
location: /tmp/airflowtmp7xkv7qws/Node_Test1sqa0v1n
[2019-07-03 12:36:32,803] {bash_operator.py:114} INFO - Running command: 
/usr/bin/node /home/developer/tryAirflowNode.js
[2019-07-03 12:36:32,810] {bash_operator.py:123} INFO - Output:
[2019-07-03 12:36:32,872] {bash_operator.py:127} INFO - (node:14702) 
[DEP0013] DeprecationWarning: Calling an asynchronous function without 
callback is deprecated.
[2019-07-03 12:36:32,875] {bash_operator.py:127} INFO - 
/tmp/airflowtmp7xkv7qws/Node_Test1sqa0v1n: line 1: 14702 Aborted                 
/usr/bin/node /home/developer/tryAirflowNode.js
[2019-07-03 12:36:32,876] {bash_operator.py:131} INFO - Command exited 
with return code 134
[2019-07-03 12:36:32,887] {__init__.py:1580} ERROR - Bash command failed
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", 
line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site- 
packages/airflow/operators/bash_operator.py", line 135, in execute
raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2019-07-03 12:36:32,889] {__init__.py:1611} INFO - Marking task as 
FAILED.`

0 个答案:

没有答案
相关问题