Question

我想触发一个simplehttpoperator，如下所示： airflow trigger_dag test_trigger --conf＆＃39; {＆＃34; name＆＃34;：＆＃34; something＆＃34;}＆＃39;

然后我使用pythonoperator python_callable通过使用kwargs [＆＃39; dag_run＆＃39;]。conf来接受参数，我想将[＆＃39; dag_run＆＃39;]。conf传递给simplehttpoperator，如何我可以做吗？有人可以帮忙吗？

cc_ = {}


def run_this_func(ds, **kwargs):
    cc_ = kwargs['dag_run'].conf
    logging.info(cc_)
    return cc_

run_this = PythonOperator(
    task_id='run_this',
    provide_context=True,
    python_callable=run_this_func,
    dag=dag)

http_task = SimpleHttpOperator(
    task_id='http_task',
    http_conn_id='test_http',
    method='POST',
    endpoint='/api/v1/function',
    data=cc_,
    headers={"Authorization": "Basic YWRtaW46MTIzNDU2", "Accept": "application/json, text/plain, */*"},
    response_check=lambda response: True if "10000" in response.content else False,
    dag=dag)

http_task.set_upstream(run_this)

Answer 1

对于任务之间的通信，您可能需要检查XCOM，https://airflow.incubator.apache.org/concepts.html#xcoms

*****更新*****
（感谢Daniel了解更多细节）下面是一些您可以尝试的代码，在SimpleHttpOperator中，您可以通过XCOM获得返回值：

http_task = SimpleHttpOperator(
    task_id='http_task',
    http_conn_id='test_http',
    method='POST',
    endpoint='/api/v1/function',
    data=json.loads("{{ task_instance.xcom_pull(task_ids='run_this', key='return_value') }}"),
    headers={"Authorization": "Basic YWRtaW46MTIzNDU2", "Accept": "application/json, text/plain, */*"},
    response_check=lambda response: True if "10000" in response.content else False,
    dag=dag)

Answer 2

感谢@Chengzhi和@Daniel。最后我写了一个自定义过滤器＆＃39; tojson＆＃39;在Jinja2 / filter.py中，因为在气流中，默认的Jinja2版本是2.8.1，Jinja2不包含名为＆＃39; tojson＆＃39;的内置过滤器。直到2.9版。

def do_tojson(value):
    value = json.JSONEncoder().encode(value)
    return value

在dag文件中，代码如下。它有效。

def run_this_func(ds, **kwargs):
    cc_ = kwargs['dag_run'].conf
    return cc_

run_this = PythonOperator(
    task_id='run_this',
    provide_context=True,
    python_callable=run_this_func,
    dag=dag)

http_task = SimpleHttpOperator(
    task_id='http_task',
    http_conn_id='test_http',
    method='POST',
    endpoint='/api/v1/task',
    data="{{ task_instance.xcom_pull(task_ids='run_this') |tojson}}",
    headers={"Authorization": "Basic YWRtaW46MTIzNDU2", "Accept": "application/json, text/plain, */*",
             "Content-Type": "application/json"},
    response_check=lambda response: True if "10000" in response.content else False,
    dag=dag)

http_task.set_upstream(run_this)

如何将参数从pythonoperator任务传递到airflow dag中的simplehttpoperator任务？

2 个答案: