气流调度程序持续运行Paused DAG

Question

Airflow网络服务器

DAG暂停：

Airflow Scheduler继续运行DAG

我全新安装了Airflow。所有暂停的示例都正常工作。即使在网络上发布PAAGED，新的样本DAG也会继续运行。 PAUSING / unPAUSING在默认示例中工作正常。 其他问题是，它每6小时安排一次。为什么要继续尝试运行DAG？

SWAT_Tutorial_01.py

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta

default_args = {'owner': 'airflow',
        'depends_on_past': False,
        'start_date': datetime(2018, 3, 16),
        'email': ['sample@scholastic.com'],
        'email_on_failure': False,
        'email_on_retry': False,
        'retries': 1,
        'retry_delay': timedelta(minutes=5),
        # 'catchup': False,
        # 'queue': 'bash_queue',
        # 'pool': 'backfill',
        # 'priority_weight': 10,
        # 'end_date': datetime(2016, 1, 1),
}

dag = DAG( 'SWAT.Tutorial_01', default_args=default_args, schedule_interval=timedelta(hours=6))

# t1, t2 and t3 are examples of tasks created by instantiating operators
t1 = BashOperator( task_id='print_date', bash_command='date', dag=dag)

t2 = BashOperator( task_id='sleep', bash_command='sleep 5', retries=3, dag=dag)
t3 = BashOperator( task_id='hello', bash_command='echo "Hello World"', retries=3, dag=dag)

t2.set_upstream(t1)
t3.set_upstream(t1)

计划程序日志（不断重复）

    File Path                                         PID  Runtime    Last Runtime    Last Run
----------------------------------------------  -----  ---------  --------------  ----------
/home/airflow/airflow/dags/SWAT_Tutorial_01.py  16930  0.00s
================================================================================
[2018-03-17 23:41:45,352] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:45,352] {jobs.py:1440} INFO - Heartbeating the executor
[2018-03-17 23:41:46,354] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:46,354] {dag_processing.py:559} INFO - Processor for /home/airflow/airflow/dags/SWAT_Tutorial_01.py finished
[2018-03-17 23:41:46,356] {dag_processing.py:627} INFO - Started a process (PID: 16932) to generate tasks for /home/airflow/airflow/dags/SWAT_Tutorial_01.py - logging into /var/log/airflow/scheduler/2018-03-17/SWAT_Tutorial_01.py.log
[2018-03-17 23:41:46,357] {jobs.py:1440} INFO - Heartbeating the executor
[2018-03-17 23:41:47,358] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:47,359] {jobs.py:1440} INFO - Heartbeating the executor
[2018-03-17 23:41:48,360] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:48,360] {dag_processing.py:559} INFO - Processor for /home/airflow/airflow/dags/SWAT_Tutorial_01.py finished
[2018-03-17 23:41:48,362] {dag_processing.py:627} INFO - Started a process (PID: 16934) to generate tasks for /home/airflow/airflow/dags/SWAT_Tutorial_01.py - logging into /var/log/airflow/scheduler/2018-03-17/SWAT_Tutorial_01.py.log
[2018-03-17 23:41:48,363] {jobs.py:1440} INFO - Heartbeating the executor

Answer 1

这里只是一些猜测。 我认为气流不会停止回填数据，即使你暂停dag，这些工作可能是回填工作。 您的代码似乎没有禁用回填（catch_up = False）