Question

我使用LocalExecutor在Airflow 1.10.11上遇到了SIGTerm错误。

[2020-09-21 10:26:51,210] {{taskinstance.py:955}} ERROR - Received SIGTERM. Terminating subprocesses.

dag任务正在执行此操作：

从SQL Server（在Windows上）向pandas数据框中读取一些数据。
然后将其写入文件（甚至没有到达此部分）。

奇怪的是，如果我限制要在查询中返回的行数（例如TOP 100），则dag成功。

如果我在本地计算机上运行python代码，它将成功。我正在使用pyodbc和sqlalchemy。仅在20或30秒后，它就不会在该行上失败：

df_query_results = pd.read_sql(sql_query, engine)

气流记录

[2020-09-21 10：26：51,210] {{helpers.py:325}}信息-向GPID xxx发送Signals.SIGTERM [2020-09-21 10：26：51,210] {{taskinstance.py:955}}错误-收到SIGTERM。终止子流程。 [2020-09-21 10：26：51,804] {{taskinstance.py:1150}}错误-任务收到SIGTERM信号

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/airflow/dags/operators/sql_to_avro.py", line 39, in execute
    df_query_results = pd.read_sql(sql_query, engine)
  File "/usr/local/lib64/python3.6/site-packages/pandas/io/sql.py", line 436, in read_sql
    chunksize=chunksize,
  File "/usr/local/lib64/python3.6/site-packages/pandas/io/sql.py", line 1231, in read_query
    data = result.fetchall()
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 1216, in fetchall
    e, None, None, self.cursor, self.context
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1478, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 1211, in fetchall
    l = self.process_rows(self._fetchall_impl())
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 1161, in _fetchall_impl
    return self.cursor.fetchall()
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 957, in signal_handler
    raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2020-09-21 10:26:51,813] {{taskinstance.py:1194}} INFO - Marking task as FAILED.

编辑：我之前想念它，但是有关于主机名的警告消息。

WARNING - The recorded hostname da2mgrl001d1.match.corp does not match this instance's hostname airflow-mg-dev.i.mct360.com

Answer 1

我有一位Linux /网络工程师的帮助。不幸的是，我不知道全部细节，但是解决的方法是它们将airflow.cfg中的hostname_callable设置更改为hostname_callable = socket:gethostname。先前已将其设置为socket:getfqdn

注意：我在解决问题的地方发现了几个不同的问题（可能是相关的）。

使用熊猫时出现气流错误-任务收到SIGTERM信号

1 个答案: