Question

我在气流中设置了一个条件任务here。它所做的就是检查是否存在hive分区。如果是，请继续执行其余任务，如果不是，请先继续添加分区，然后再继续。条件检查任务可能失败或成功，两者都可以。但是，我为dag设置了寻呼机职责电子邮件通知，因为我想知道下游任务何时失败。如何将该特定条件任务的失败通知静音，以便我不会在寻呼机上发生误报？

Answer 1

email_on_failure和on_failure_callback等是任务（操作员）级别参数。它们继承自DAG对象，传递给DAG的default_args值，但您也可以在初始化时覆盖它们。

YourOperator(task_id='task1', dag=dag, email_on_failure=None, on_failure_callback=None, ...)

这是源代码，当任务失败时，气流如何处理这些回调，让你更清楚它是如何工作的。

def handle_failure(self, error, test_mode=False, context=None):
        self.log.exception(error)
        task = self.task
        session = settings.Session()
        self.end_date = datetime.utcnow()
        self.set_duration()
        Stats.incr('operator_failures_{}'.format(task.__class__.__name__), 1, 1)
        Stats.incr('ti_failures')
        if not test_mode:
            session.add(Log(State.FAILED, self))

        # Log failure duration
        session.add(TaskFail(task, self.execution_date, self.start_date, self.end_date))

        # Let's go deeper
        try:
            # Since this function is called only when the TI state is running,
            # try_number contains the current try_number (not the next). We
            # only mark task instance as FAILED if the next task instance
            # try_number exceeds the max_tries.
            if task.retries and self.try_number <= self.max_tries:
                self.state = State.UP_FOR_RETRY
                self.log.info('Marking task as UP_FOR_RETRY')
                if task.email_on_retry and task.email:
                    self.email_alert(error, is_retry=True)
            else:
                self.state = State.FAILED
                if task.retries:
                    self.log.info('All retries failed; marking task as FAILED')
                else:
                    self.log.info('Marking task as FAILED.')
                if task.email_on_failure and task.email:
                    self.email_alert(error, is_retry=False)
        except Exception as e2:
            self.log.error('Failed to send email to: %s', task.email)
            self.log.exception(e2)

        # Handling callbacks pessimistically
        try:
            if self.state == State.UP_FOR_RETRY and task.on_retry_callback:
                task.on_retry_callback(context)
            if self.state == State.FAILED and task.on_failure_callback:
                task.on_failure_callback(context)
        except Exception as e3:
            self.log.error("Failed at executing callback")
            self.log.exception(e3)

        if not test_mode:
            session.merge(self)
        session.commit()
        self.log.error(str(error))

https://airflow.apache.org/_modules/airflow/models.html#BaseOperator

如何使特定气流任务失败？

1 个答案: