Question

有人知道在bash提示符下运行airflow test时是否有办法设置dag_run.conf参数？

例如，我已从官方气流存储库下载example_trigger_target_dag，并且我想测试run_this任务。通常我会做以下事情：

~/$ airflow test example_trigger_target_dag run_this '2018-01-01'

但是运行它会产生错误：

--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2018-05-02 10:50:01,154] {models.py:1342} INFO - Executing <Task(PythonOperator): run_this> on 2018-01-01 00:00:00
[2018-05-02 10:50:01,262] {models.py:1417} ERROR - 'NoneType' object has no attribute 'conf'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1374, in run
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/dist-packages/airflow/operators/python_operator.py", line 80, in execute
    return_value = self.python_callable(*self.op_args, **self.op_kwargs)
  File "/home/annalect/uk_ds_airflow/dags/playpen/example_trigger_target_dag.py", line 56, in run_this_func
    print("Remotely received value of {} for key=message".format(kwargs['dag_run'].conf['message']))
AttributeError: 'NoneType' object has no attribute 'conf'

我已经开始使用task_params参数但是我的语法错误或者它没有达到我之后的效果，因为它会产生与上面相同的错误：

~/$ airflow test --task_params '{"kwargs": {"dag_run": {"conf": {"message": "Hey world"}}}}' example_trigger_target_dag run_this '2018-01-01'

[2018-05-02 11:10:58,065] {models.py:1441} INFO - Marking task as FAILED.
[2018-05-02 11:10:58,070] {models.py:1462} ERROR - 'NoneType' object has no attribute 'conf'

所以有人知道如何测试依赖于dag_run.conf值的任务吗？

谢谢！

Answer 1

override func viewDidLoad() { super.viewDidLoad() self.navigationItem.leftBarButtonItem = UIBarButtonItem(title: "back", style: .plain, target: self, action: #selector(self.back)) } @objc func back(){ //Create and set up the animation let transition = CATransition() transition.duration = 0.4 transition.type = kCATransitionMoveIn transition.subtype = kCATransitionFromRight//animates from right to left self.navigationController?.view.layer.add(transition, forKey: nil)//adds the animation self.navigationController?.popViewController(animated: true) }命令没有--conf选项，但您可以通过将参数传递给任务airflow test来解决此问题。

在callable中，如果设置了python_callable，则可以检索参数以构建虚拟kwargs['test_mode']对象，如下所示：

DagRun

要测试from airflow.models import DagRun ... def run_this_func(ds, **kwargs): if kwargs['test_mode']: kwargs['dag_run'] = DagRun(conf=kwargs['params']) print("Remotely received value of {} for key=message".format(kwargs['dag_run'].conf['message']))，只需执行以下操作：

example_trigger_target_dag

你会得到：

airflow test example_trigger_target_dag test_trigger_dagrun "2018-01-01" -tp '{"message":"Hello world"}'

现在，您可以编写装饰器，而不是将测试代码放入任务中。此外，由于我们只使用Remotely received value of Hello world for key=message的{{1}}属性，因此我们也可以使用conf。最后，为了避免在查找DagRun时出现潜在的关键错误，我们可以将SimpleNamespace与默认值一起使用。

kwargs

在调用airflow测试时设置dag_run.conf参数

1 个答案: