Question

谁能建议我如何在“ myfunction”方法中将dag-context作为参数传递。

dag_1.py
from datetime import datetime
from airflow import DAG
from utils.myutils import generate_file_list

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2017, 7, 17),
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'dataflow_default_options': {
        'project': Variable.get('project_id'),
    }
}

input_files = generate_file_list( context ) # facing issue in this line, My requirement is to pass dag-context within this function. 


with DAG('dag_1', schedule_interval=None, catchup=False, default_args=default_args) as dag:

    task_1=PythonOperator(
        task_id='task_1',
        python_callable=some_callable,
        provide_context=True
    )

    task_2=PythonOperator(
        task_id='task_3',
        python_callable=some_callable,
        provide_context=True
    )

   task_1 >> task_2

   for file_name, file_path in input_files.items():

     task_3=PythonOperator(
            task_id='task_1',
            python_callable=some_callable,
            provide_context=True
      )


-----------------------------------------------
myutils.py

import logging

def generate_file_list(**kwargs):
    directory_name=kwargs["dag_run"].conf["directory_name"]
    directory_id=kwargs["dag_run"].conf["directory_id"]
    return file_list(directory_name, directory_id)

def file_list(directory_name, directory_id):
    file_name_dic = { }
    src_dir='gs://bucket_name/folder'
    ...............
    ....................
    return file_name_dic

----------------------------------------------

In my case dag_1 (DAG) is triggered through a curl command and below payload is sent to dag:
conf = {
        'directory_name': args.directory_name,
        'directory_id': args.directory_id }

注意：directory_id和directory_name将使用curl命令作为运行时参数传递。

我正在编写一个dag，并想在“ my-fucntion”下访问dag-context以检索有效负载。我的函数方法将返回具有文件名和file_path的python字典对象。根据my函数的返回值，某些任务将循环执行

有什么方法可以将上下文或有效载荷传递给myFucntion？

0 个答案: