Question

我在GCP Airflow上有一个DAG，其任务如下：

with DAG(dag_name, schedule_interval='0 6 * * *', default_args=default_dag_args) as dag:

    notify_start = po.PythonOperator(
        task_id = 'notify-on-start',
        python_callable = slack,
        op_kwargs={'msg': slack_start}
    )

    create_dataproc_cluster = d.create_cluster(default_dag_args['cluster_name'], service_account, num_workers)

    [assorted dataproc tasks]

    notify_on_fail = po.PythonOperator(
        task_id = 'notify-on-task-failure',
        python_callable = slack,
        op_kwargs={'msg': slack_error, 'err': True},
        trigger_rule = trigger_rule.TriggerRule.ONE_FAILED
    )

    delete_cluster = d.delete_cluster(default_dag_args['cluster_name'])

    notify_finish = po.PythonOperator(
        task_id = 'notify-on-completion',
        python_callable = slack,
        op_kwargs={'msg': slack_finish},
        trigger_rule = trigger_rule.TriggerRule.ALL_DONE
    )

    notify_start >> create_dataproc_cluster >> [assorted dataproc tasks >> delete_cluster >> notify_on_fail >> notify_finish

我面临的问题是，即使有notify_on_fail触发规则，如果dataproc任务之一失败，ONE_FAILED任务也不会触发。相反，它会旋转群集并发送所有清晰的消息（notify_finish）我的任务顺序错误，还是其他错误？

Answer 1

根据您的期望，我认为DAG应该是这样的

notify_start >> create_dataproc_cluster >> [assorted dataproc tasks]
[assorted dataproc tasks] >> notify_on_fail >> delete_cluster
[assorted dataproc tasks] >> notify_finish >> delete_cluster

满足触发规则ONE_FAILED时不触发气流任务

1 个答案: