Question

我正在研究这个气流dag文件，以对XCOM进行一些测试，但不确定如何在python运算符之间使用它。有人可以帮助使用XCOM push和pull函数编写如何在python运算符之间传递消息的逻辑。以下是我正在处理的dag文件...

问题是如何将消息从每个任务传递到另一个任务

@dag_factory
def create_dag():
    with DAG(
        dag_id="DSStest",
        default_args=default_args,
        schedule_interval=timedelta(1),
    ) as dag:
        # Define operators here, for example:

        output_file = path_in_workspace("testout")

        rscript_file = path_in_workspace("rtest2.R ")

        bcmd1 = "downloading some file here..."

        t0 = PythonOperator(
            task_id="start",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 0],
        )

        t1 = PythonOperator(
            task_id="job1",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 1],
        )

        t2 = PythonOperator(
            task_id="job2",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 2],
        )

        t10 = PythonOperator(
            task_id="job10",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 10],
        )

        t20 = BashOperator(
            task_id="job20",
            bash_command=bcmd1,
            queue={
                "worker_type": "png-bash-worker",
                "request_memory": "1G",
                "request_cpu": 1,
            },
        )

        # Define dependencies between operators here, for example:

        t0 >> t1
        t0 >> t2
        t1 >> t10
        t2 >> t10
        t10 >> t11
        t11 >> t20

        return dag  # Do not change this

Answer 1

我建议看一下这个示例，它显示/解释了有关xcoms和PythonOperators的所有内容：
example_xcom.py

官方气流页面还详细解释了xcom：
official documentation

如何使用python运算符在气流dag文件中使用xCom？

1 个答案: