气流+芹菜+花-任务名称不是Python操作员的任务ID

时间:2019-03-17 19:23:51

标签: airflow airflow-scheduler flower

我当前正在使用CeleryExecutor提交DAG,并且正在AWS的同一容器中运行Web服务器,调度程序和工作程序。当我提交要运行的DAG并且其任务在Redis队列中(使用Elasticache)时,Celery工作人员将执行任务,但名称仅为airflow.executors.celery_executor.execute_command。我如何使它们成为args?enter image description here

当前,这些是airflow.cfg的重要部分:

# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor
executor = CeleryExecutor

[celery]
# This section only applies if you are using the CeleryExecutor in
# [core] section above

# The app name that will be used by celery
celery_app_name = airflow.executors.celery_executor

在我们的DAG文件中:

dag = DAG(
    'foo_dag',
    schedule_interval='0 2-23 * * *',
    catchup=False,
    default_args=default_args
)

t1 = PythonOperator(
    task_id="bar_task",
    provide_context=True,
    python_callable=bar_task,
    op_kwargs={"emr_cluster_id": emr_cluster_id},
    task_concurrency=1,
    dag=foo_dag
)

0 个答案:

没有答案