Question

我正在测试气流（版本1.10.5和python 3）DAG，并注意到，每当我尝试清除任务时，它都会导致蘑菇状云，并指出SSLContext对象无法序列化。

无论任务的状态如何，这种情况似乎都会发生-但是，如果我手动将其标记为“失败”，然后再清除，则似乎可以正常工作。

不确定发生了什么，对此深表感谢！谢谢

踪迹：

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 258, in decorated_view
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 281, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 328, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 1261, in clear
    include_upstream=upstream)
  File "/usr/local/lib/python3.6/site-packages/airflow/models/dag.py", line 1051, in sub_dag
    for t in regex_match + also_include}
  File "/usr/local/lib/python3.6/site-packages/airflow/models/dag.py", line 1051, in <dictcomp>
    for t in regex_match + also_include}
  File "/usr/local/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/usr/local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 626, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/usr/local/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.6/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle SSLContext objects

-编辑-

所以我做了更多的挖掘工作，我认为这是因为DAG中的一个任务使用googleads对其API进行了调用。该库使用SSLContext，似乎是导致此问题的原因（https://github.com/googleads/googleads-python-lib/blob/f070075a081cf2b32fd3d3b1b34b3e3770858c52/googleads/common.py#L440）

不确定，但是气流应用程序代码中是否有一些可以克服此问题的代码？

Answer 1

您是否在DAG中的其中一个运算符的构造函数中设置了挂钩或连接？

如果是，请尝试直接在execute方法的开头设置此挂钩/连接，并将其从__init__方法中删除。

PostgresOperator 是一个很好的例子。 https://github.com/apache/airflow/blob/master/airflow/operators/postgres_operator.py#L65-L67

清除气流任务会导致TypeError错误（蘑菇云）：无法腌制SSLContext对象

1 个答案: