Question

我正在使用Airflow 1.8.1。我有一个DAG，我相信我计划每5分钟运行一次，但它没有这样做：

忽略2个成功运行的DAG运行，这些运行是手动触发的。

我查看该DAG的调度程序日志，然后看到：

[2019-04-26 22:03:35,601] {jobs.py:343} DagFileProcessor839 INFO - Started process (PID=5653) to work on /usr/local/airflow/dags/retrieve_airflow_artifacts.py
[2019-04-26 22:03:35,606] {jobs.py:1525} DagFileProcessor839 INFO - Processing file /usr/local/airflow/dags/retrieve_airflow_artifacts.py for tasks to queue
[2019-04-26 22:03:35,607] {models.py:168} DagFileProcessor839 INFO - Filling up the DagBag from /usr/local/airflow/dags/retrieve_airflow_artifacts.py
[2019-04-26 22:03:36,083] {jobs.py:1539} DagFileProcessor839 INFO - DAG(s) ['retrieve_airflow_artifacts'] retrieved from /usr/local/airflow/dags/retrieve_airflow_artifacts.py
[2019-04-26 22:03:36,112] {jobs.py:1172} DagFileProcessor839 INFO - Processing retrieve_airflow_artifacts
[2019-04-26 22:03:36,126] {jobs.py:566} DagFileProcessor839 INFO - Skipping SLA check for <DAG: retrieve_airflow_artifacts> because no tasks in DAG have SLAs
[2019-04-26 22:03:36,132] {models.py:323} DagFileProcessor839 INFO - Finding 'running' jobs without a recent heartbeat
[2019-04-26 22:03:36,132] {models.py:329} DagFileProcessor839 INFO - Failing jobs without heartbeat after 2019-04-26 21:58:36.132768
[2019-04-26 22:03:36,139] {jobs.py:351} DagFileProcessor839 INFO - Processing /usr/local/airflow/dags/retrieve_airflow_artifacts.py took 0.539 seconds
[2019-04-26 22:04:06,776] {jobs.py:343} DagFileProcessor845 INFO - Started process (PID=5678) to work on /usr/local/airflow/dags/retrieve_airflow_artifacts.py
[2019-04-26 22:04:06,780] {jobs.py:1525} DagFileProcessor845 INFO - Processing file /usr/local/airflow/dags/retrieve_airflow_artifacts.py for tasks to queue
[2019-04-26 22:04:06,780] {models.py:168} DagFileProcessor845 INFO - Filling up the DagBag from /usr/local/airflow/dags/retrieve_airflow_artifacts.py
[2019-04-26 22:04:07,258] {jobs.py:1539} DagFileProcessor845 INFO - DAG(s) ['retrieve_airflow_artifacts'] retrieved from /usr/local/airflow/dags/retrieve_airflow_artifacts.py
[2019-04-26 22:04:07,287] {jobs.py:1172} DagFileProcessor845 INFO - Processing retrieve_airflow_artifacts
[2019-04-26 22:04:07,301] {jobs.py:566} DagFileProcessor845 INFO - Skipping SLA check for <DAG: retrieve_airflow_artifacts> because no tasks in DAG have SLAs
[2019-04-26 22:04:07,307] {models.py:323} DagFileProcessor845 INFO - Finding 'running' jobs without a recent heartbeat
[2019-04-26 22:04:07,307] {models.py:329} DagFileProcessor845 INFO - Failing jobs without heartbeat after 2019-04-26 21:59:07.307607
[2019-04-26 22:04:07,314] {jobs.py:351} DagFileProcessor845 INFO - Processing /usr/local/airflow/dags/retrieve_airflow_artifacts.py took 0.538 seconds

一遍又一遍。我已经将其与另一台服务器上的DAG进行了比较，因此，我知道会有额外的日志记录表明DAG已通过计划触发，因此该日志文件中没有此类记录。

以下是我DAG时间表的定义方式：

args = {
    'owner': 'airflow',
    'start_date': (datetime.datetime.now() - datetime.timedelta(minutes=5))
}

dag = DAG(
    dag_id='retrieve_airflow_artifacts', default_args=args,
    schedule_interval="0,5,10,15,20,25,30,35,40,45,50,55 * * * *")

有人可以帮我弄清楚为什么我的DAG不能运行的原因，因为我看上去过高和过低，无法弄清楚。

Answer 1

如果我不得不猜测，我会说您的start_date引起了您一些问题。

更改您的arg以使其具有静态起始位置，并防止其以过去的间隔运行：

args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2019, 4, 27) #year month day
}

此外，为了使其更易于阅读，请将DAG参数更改为（相同功能）：

dag = DAG(
    dag_id='retrieve_airflow_artifacts', 
    default_args=args,
    schedule_interval="*/5 * * * *"
)

这应该允许调度程序将其提取！

通常建议不要动态设置起始日期。

取自Airflow FAQ：

我们建议不要将动态值用作开始日期，尤其是 datetime.now（），因为它可能会令人困惑。任务被触发一旦周期结束，理论上@hourly DAG就永远不会 now（）沿现在移动到一个小时后。

关于此的另一个SO问题：why dynamic start dates cause issues

预定的DAG未运行。我如何诊断问题？

1 个答案: