Question

我想创建自己的自定义基本运算符，并具有在成功/失败/重试时向度量指标发送度量的额外功能

我创建了一个监控类

class Monitoring
    def __init__(self,
             metrics,*args, **kwargs):
        self.metrics = builtin_metrics + metrics if metrics is not None else builtin_metrics
    def send_metric(self,status, context):
       CODE TO SEND TO STACKDRIVER
    def _on_success_callback(self,context):
       self.send_metric('SUCCESS, context)

class CustomBaseOperator(Monitoring, BaseOperator):
      def __init__(self,  *args, **kwargs):
         Monitoring.__init__(self, *args, **kwargs)
         BaseOperator.__init__(self,
                          on_success_callback=self._on_success_callback
                          *args, **kwargs)

我是这样设计的，以便所有监视功能都在其自己的类中处理，并且如果我们想向基本运算符添加更多功能，可以通过从其他类继承来完成

这在python中可以正常工作，但有时在气流中也可以，但是当我重新启动服务器（本地）时通常会失败：

File "/Users/nirben/anaconda3/envs/airflow/lib/python3.6/site-packages/airflow/www/views.py", line 1335, in recurse_nodes
visited.add(task)
File "/Users/nirben/anaconda3/envs/airflow/lib/python3.6/site-packages/airflow/models.py", line 2517, in __hash__
hash(val)
File "/Users/nirben/anaconda3/envs/airflow/lib/python3.6/site-packages/airflow/models.py", line 2517, in __hash__
hash(val)
File "/Users/nirben/anaconda3/envs/airflow/lib/python3.6/site-packages/airflow/models.py", line 2517, in __hash__
hash(val)
[Previous line repeated 314 more times]
RecursionError: maximum recursion depth exceeded

我需要on_success_callback方法为实例，因为它们记录特定于给定任务/数据的数据

气流部分是否有阻止这种情况的限制？

气流在基本运算符中定义on_success_callback

0 个答案: