气流-任务之间的变量

Question

如何在DAG级别创建变量并将其传递给多个任务？

例如：

cluster_name = 'data-' + datetime.now().strftime("%Y-%m-%d-%H-%M-%S-%f")

我必须在所有任务中使用以上变量cluster_name。但我看到价值不断变化。我不想使用xcom。请指教

Answer 1

This value will change all the time because the DAG definition is being parsed repeatedly by the scheduler/webserver/workers, and datetime.now() will return different values every time it is parsed.

I highly recommend against using dynamic task names.

The date is already part of a task in the sense that the execution date is part of what makes each run of the task unique.

Each task instance can be identified by: dag_id + task_id + execution_date

To uniquely identify the tasks, use these things instead of bundling the date inside the name.

Answer 2

您可以将其存储在Airflow Variable中，并且所有任务都可以访问它。请注意，每次您查找变量时，这都是数据库调用。