Question

从UI清除Dag实例时，我得到以下AirflowException，但我可以从cli清除它我最近将气流更新为1.9.0

我在models.py里面添加了以下try除了块，以查找哪个参数不可序列化

try:
    setattr(result, k, copy.deepcopy(v, memo))
except TypeError as te:
    raise AirflowException('Failed to deep_copy the value %s: %s'%(v, te))

导致此问题的任务是自定义文件传感器，该传感器继承自BaseOperator

我已完成以下记录：

在配置文件夹中创建 init .py和log_config.py文件
在log_config.py中复制airflow_local_settings.py的内容
在log_config内将DEFAULT_LOGGING_CONFIG重命名为LOGGING_CONFIG
在airflow.csfg文件中添加logging_level = INFO和logging_config_class = log_config.LOGGING_CONFIG

BaseSensor

class BaseSensor(BaseOperator):
    @apply_defaults
    def __init__(self, want_skip_on_fail=False, *args, **kw):
        super(BaseSensor, self).__init__(*args, **kw)
        self.want_skip_on_fail = want_skip_on_fail

    def execute(self, context):
        ti = context['task_instance']
        if not self.poke(context):
            if (self.want_skip_on_fail and ti.try_number >= self.retries):
                raise AirflowSkipException('failed poke - skipping.')
            else:
                raise AirflowException('failed poke - aborting.')
        ti.log.info("Success criteria met. Exiting.")

    def poke(self, context):
        raise AirflowException('You must override BaseSensor::poke().')

并且在FileSensor Operator中，我删除了几乎所有代码，以便它始终从poke函数返回true并打印日志消息

错误消息

Jun 25 22:27:46 bcpmrstbssapp1 airflow: File "/opt/nio/lib/airflow/airflow/models.py", line 2440, in __deepcopy__
Jun 25 22:27:46 bcpmrstbssapp1 airflow: raise AirflowException('Failed to deep_copy the value %s: %s'%(v, te))
Jun 25 22:27:46 bcpmrstbssapp1 airflow: AirflowException: Failed to deep_copy the value <logging.Logger object at 0x7fe34cd03a10>: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

Jun 26 00:50:52 bcpmrstbssapp1 airflow: 10.46.1.124 - - [26/Jun/2019:00:50:52 +0700] "GET /admin/airflow/clear?task_id=test_custom_feeds_unit.file_rsync.dw_queue.flows&dag_id=test_custom_feeds_unit_dna&future=false&past=false&upstream=false&downstream=true&recursive=true&execution_date=2019-06-25T11:30:00&origin=http%3A%2F%2F10.20.0.152%3A8142%2Fadmin%2Fairflow%2Ftree%3Fdag_id%3Dtest_custom_feeds_unit_dna HTTP/1.1" 500 4075 "http://10.20.0.152:8142/admin/airflow/tree?dag_id=test_custom_feeds_unit_dna" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

有人遇到过类似的问题吗？

更新

从源代码调试models.py后，我发现无法复制的参数是：

self._log = logging.getLogger（“ airflow.task”）

解决方法：在我的自定义运算符中，我设置了self._log = None

从UI清除dag但从cli清除dag时出现气流异常

0 个答案: