Question

在所有父任务都成功然后仅下一个任务被触发的情况下，我在工作流程中使用触发规则=“ all_sucess”。但是没有发生同样的事情。

工作流程：

任务1 >>任务2a，任务2b，任务2c（并行运行）>>任务c

方案1-任务2a之一进入up_for_retry，即使任务c被执行

方案2-任务2a之一处于运行状态，任务c仍被执行

注意-在所有地方，trigger_rule都是all_success

在理想情况下，直到所有任务2a，2b，2c成功完成后，才应触发任务C。

query_template_dict = {
    'partner_list' = ['val1', 'val2']
    'google_project': 'project_name',
    'queries': {
        'layer3': {
            'template':             'temp.sql'
        }
    },
    'applicable_tasks': {
        'val1': {
            'table_layer3': ['activity']
        },
        'val2': {
            'table_layer3': ['activity'],
        }

    }
}


for partner in query_template_dict['partner_list']:
    # Loop over applicable report queries for a partner
    applicable_tasks = query_template_dict['applicable_tasks'][partner].keys()
    for task in applicable_tasks:
                                              
        query_params=[
        {
                "name":                 "col1",
                "parameterType":        { "type": "STRING" },
                "parameterValue":       { "value": col1}
        }
        ]
        
        run_bq_cmd = BigQueryOperator (
                        task_id                                 =partner + '-' + task
                        trigger_rule                            ='all_success',
                        allow_large_results                     =True,
                        dag=dag
                        )
        # Creating dependency on previous tasks
        run_dummy >> run_bq_cmd
        
        for sub_tasks in query_template_dict['applicable_tasks'][partner][task]:
        
            run_sub_task = BashOperator(task_id = partner+ '_' + task + '_' + sub_tasks,
                                        bash_command = bash_command,
                                        trigger_rule= 'all_success',
                                        dag = dag
                                       )
            run_bq_cmd >> run_sub_task                           
            bash_command    = <some bash command>       
            end_task = BashOperator( task_id =      'end_task',
                                     bash_command=  bash_command,
                                     trigger_rule=  'all_success',
                                     dag=           dag
                                    )
            # Creating dependency on previous tasks
            run_sub_task >> end_task

即使多个父任务（运行子任务）尚未完成，结束任务也会被调用。

有人可以帮忙吗

Answer 1

您是否已经在Airlfow UI中进行了检查，是否可以从图形的角度查看子任务和最终任务的正确链接？

我认为您没有很好地设置依赖性，并且似乎在实例化其父任务的同一循环中实例化了end_task。尝试做类似的事情：

for partner in query_template_dict['partner_list']:
    
    # your previous code ...
    # ...
    # ...
    
    bash_command    = <some bash command>       
    end_task = BashOperator( task_id =      'end_task',
                             bash_command=  bash_command,
                             trigger_rule=  'all_success',
                             dag=           dag
                            )
    
    for sub_tasks in query_template_dict['applicable_tasks'][partner][task]:
    
        run_sub_task = BashOperator(task_id = partner+ '_' + task + '_' + sub_tasks,
                                    bash_command = bash_command,
                                    trigger_rule= 'all_success',
                                    dag = dag
                                   )
                       
        # Create dependency on previous and next tasks
        run_bq_cmd >> run_sub_task >> end_task

All_Success触发规则未按预期运行

1 个答案: