气流启动多个并发的通用任务

时间:2018-09-04 16:05:44

标签: airflow google-cloud-composer

尝试在Cloud Composer上同时完成一些任务:

arr = {}
for i in xrange(3):
    print("i: " + str(i))
    command_formatted = command_template.format(str(i))
    create_training_instance = bash_operator.BashOperator(
        task_id='create_training_instance',
        bash_command=command_formatted)
    arr[i] = create_training_instance
    start_training.set_downstream(arr[i])  

出现以下错误:

  

破损的DAG:[/ home / airflow / gcs / dags / scale_simple.py]依赖性   ,已经create_training_instance   已注册

2 个答案:

答案 0 :(得分:1)

task_id对于单个任务应该始终是唯一的。因此,您可以将create_training_instance_{}.format(i)用作task_id

答案 1 :(得分:0)

您还需要参数化您的任务ID,例如, task_id ='create_training_instance'->'create_traiing_instance-{}'。format(i)