Question

我当前正在使用Big-query Operator see here。我需要明确发送用户生成的JobId参数。

我知道此操作符默认情况下会创建自己的生成的JobId，但是我有兴趣重写此行为，很遗憾，此操作符不接受JobId作为参数。

我试图通过api_resource_configs传递这样的参数，并收到以下消息（请注意，我的JobId参数被忽略了）：

[2019-02-12 14:01:09,402] {base_task_runner.py:98} INFO - Subtask: /usr/local/lib/airflow/airflow/models.py:2159: PendingDeprecationWarning: Invalid arguments were passed to BigQueryOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were:
[2019-02-12 14:01:09,403] {base_task_runner.py:98} INFO - Subtask: *args: ()
[2019-02-12 14:01:09,403] {base_task_runner.py:98} INFO - Subtask: **kwargs: {'api_resource_configs': {'jobReference': {'jobId': 'goodfornothing'}}}
[2019-02-12 14:01:09,403] {base_task_runner.py:98} INFO - Subtask:   category=PendingDeprecationWarninghere any version of this operator that gets that argument? can i some how modify the operator to accept this argument?

提前谢谢！！！！

Answer 1

自从Airflow v1.10.2（code）中引入了api_resource_configs以来，Cloud Composer尚不支持该版本。您可以考虑将最新的BigQueryOperator代码作为您自己的自定义运算符复制到您的项目中。

修改BigQuery运算符以显式传递JobId参数AirFlow运算符

1 个答案: