我当前正在将kubernetes executor与Airflow一起使用,并且气流中的所有运算符均正常运行,但是包含kubernetes pod运算符的气流中安排的所有dag均失败.airflow UI上未显示任何日志。我只能在调度程序窗格上找到“跟随”错误日志。
[2019-11-26 13:32:44,874] {kubernetes_executor.py:362} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 had an event of type MODIFIED
[2019-11-26 13:32:44,881] {kubernetes_executor.py:398} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 Failed
[2019-11-26 13:32:44,907] {kubernetes_executor.py:362} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 had an event of type DELETED
[2019-11-26 13:32:44,915] {kubernetes_executor.py:398} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 Failed
[2019-11-26 13:32:46,850] {kubernetes_executor.py:506} INFO - Attempting to finish pod; pod_id: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9; stat
e: failed; labels: {u'try_number': '1', u'execution_date': '2019-11-26T13_32_35.285788_plus_00_00', u'airflow-worker': '45390e71-9831-4ea4-9077-a2753eff16ae', u'tas
k_id': 'passing-task', u'dag_id': 'kubernetes-operator'}
[2019-11-26 13:32:46,854] {kubernetes_executor.py:625} INFO - Checking 1 task instances.
[2019-11-26 13:32:46,855] {kubernetes_executor.py:635} INFO - Found matching task kubernetes-operator-passing-task (2019-11-26 13:32:35.285788+00:00) with current s
tate of queued
[2019-11-26 13:32:46,858] {kubernetes_executor.py:506} INFO - Attempting to finish pod; pod_id: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9; stat
e: failed; labels: {u'try_number': '1', u'execution_date': '2019-11-26T13_32_35.285788_plus_00_00', u'airflow-worker': '45390e71-9831-4ea4-9077-a2753eff16ae', u'tas
k_id': 'passing-task', u'dag_id': 'kubernetes-operator'}
[2019-11-26 13:32:46,862] {kubernetes_executor.py:625} INFO - Checking 1 task instances.
[2019-11-26 13:32:46,863] {kubernetes_executor.py:635} INFO - Found matching task kubernetes-operator-passing-task (2019-11-26 13:32:35.285788+00:00) with current s
tate of queued
[2019-11-26 13:32:46,866] {kubernetes_executor.py:819} INFO - Changing state of ((u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 3
5, 285788, tzinfo=tzlocal()), 1), u'failed', 'kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9', '688810') to failed
[2019-11-26 13:32:46,871] {kubernetes_executor.py:855} INFO - Deleted pod: (u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 35, 285
788, tzinfo=tzlocal()), 1)
[2019-11-26 13:32:46,872] {kubernetes_executor.py:819} INFO - Changing state of ((u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 3
5, 285788, tzinfo=tzlocal()), 1), u'failed', 'kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9', '688811') to failed
[2019-11-26 13:32:46,876] {kubernetes_executor.py:855} INFO - Deleted pod: (u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 35, 285
788, tzinfo=tzlocal()), 1)
[2019-11-26 13:32:50,854] {scheduler_job.py:1288} INFO - Executor reports execution of kubernetes-operator.passing-task execution_date=2019-11-26 13:32:35.285788+00
:00 exited with status failed for try_number 1
[2019-11-26 13:32:50,859] {scheduler_job.py:1304} ERROR - Executor reports task instance <TaskInstance: kubernetes-operator.passing-task 2019-11-26 13:32:35.285788+
00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
[2019-11-26 13:32:50,860] {dagbag.py:92} INFO - Filling up the DagBag from /root/airflow/dags/kubernetes_pod_operator_test.py
[2019-11-26 13:32:50,866] {taskinstance.py:1058} ERROR - Executor reports task instance <TaskInstance: kubernetes-operator.passing-task 2019-11-26 13:32:35.285788+0
0:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
None
[2019-11-26 13:32:50,868] {taskinstance.py:1089} INFO - Marking task as FAILED.
我的Dag的代码是:
import airflow
from airflow import DAG
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'start_date': airflow.utils.dates.days_ago(2),
}
dag = DAG(
dag_id='kubernetes-operator',
default_args=default_args,
schedule_interval=None,
)
task1 = KubernetesPodOperator(namespace='default',
image="Python:3.6",
cmds=["Python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
name="passing-test",
task_id="passing-task",
get_logs=True,
dag=dag
)