Question

我要实现的目标是这样的：

从UI或预先安排的运行DAG
正在不断检查队列（Redis队列）的传感器操作员
当队列接收到数据时，传感器会弹出数据并将数据推送到xcom中，以供其他操作员读取。
传感器返回true后，将触发另一端的PythonOperator。
然后传感器再次循环运行。

第一个和第二个已实现，但在成功操作后未调用PythonOperator。

这是我的传感器代码：

class DispatcherQueueSensor(BaseSensorOperator):
"""
simple Redis queue sensor to look up in the sensor.
"""

@apply_defaults
def __init__(self, queue_name, *args, **kwargs):
    super(DispatcherQueueSensor, self).__init__(*args, **kwargs)
    self.queue_name = queue_name

def poke(self, context):
    changes = []
    queue = RedisQueue(self.queue_name)

    if queue.empty():
        print("Queue is empty")
        return False

    if queue.qsize() < 1:
        print("Queue doesnot have a significant number of changes request.")
        return False
    else:
        print("Connecting to queue.......")
        value = queue.get()
        changes.append(value)
        print("CHANGE: ", value)
        context["ti"].xcom_push("change_event", value)
        return True

我的DAG如下：

import sys

sys.path.insert(0, "plugins")

from airflow import DAG
from airflow.operators import PythonOperator
from airflow.operators.dagrun_operator import TriggerDagRunOperator
from datetime import datetime, timedelta

from dispatcher_plugin.operators.sensors import DispatcherQueueSensor


default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "start_date": datetime(2018, 9, 26),
    "email": ["example@example.com"],
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=1),
}

dag = DAG(
    "dispatcher.change_queue_listener",
    default_args=default_args,
    schedule_interval="* * * * *",
)


queue_sensor_task = DispatcherQueueSensor(
    task_id="check_dag_event_for_change",
    queue_name="changes",
    dag=dag,
    poke_interval=60,
)


def trigger_queue_reader_dag(**context):
    print("=====here====")
    event = context["ti"].xcom_pull(
        task_ids="check_dag_event_for_change", key="change_event"
    )
    if event is not None:
        print(event)
        trigger_dag_id = "msdf.change-applier"
        print(trigger_dag_id)
        trigger_task = TriggerDagRunOperator(
            task_id="trigger_change_applier",
            trigger_dag_id=trigger_dag_id,
            python_callable=lambda: True,
            params={},
            dag=dag,
        )
        trigger_task.set_upstream(select_queue_tsk)


select_queue_tsk = PythonOperator(
    task_id="trigger_queue_reader_dag",
    python_callable=trigger_queue_reader_dag,
    provide_context=True,
    dag=dag,
)

queue_sensor_task >> select_queue_tsk

可能是什么问题？任何见解都会受到赞赏。

Ariflow传感器未触发python运算符

0 个答案: