Question

我正在使用气流运行工作流程DAG。我设置的开始日期是2014年5月26日，schedule_interval是没有结束日期的1天，它使用的是Celery Executor。 Airflow从给定日期开始运行我的任务，并按预期从开始日期开始每天递增。

但问题是，它将从开始日期开始计划并运行dag仅16天，然后停止。我必须重新启动调度程序服务，以便安排下一组dag并运行。

我修改了下面的属性，在airflow.cfg中将它们设置得更高，但它仍然不起作用，

digraph "graph_window" {
 node [
  style = "radial"
  fillcolor = "white"
  shape = "ellipse"
  label = ""
 ]
 edge [
  arrowhead = "vee"
 ]
 subgraph "1" {
   rank = "same"
   rankdir = "LR"
  subgraph "['Cecilia Cavendish-Bentinck','Claude Bowes-Lyon']" {
   "'Cecilia Cavendish-Bentinck'" [
     fillcolor = "white:red"
     label = "Cecilia Cavendish-Bentinck"
   ]
   "'Claude Bowes-Lyon'" [
     fillcolor = "white:green"
     label = "Claude Bowes-Lyon"
   ]
  }
  subgraph "['George V','Princess Mary of Teck']" {
   "'George V'" [
     fillcolor = "white:green"
     label = "George V"
   ]
   "'Princess Mary of Teck'" [
     fillcolor = "white:red"
     label = "Princess Mary of Teck"
   ]
  }
 }
 subgraph "2" {
   rank = "same"
   rankdir = "LR"
  subgraph "['George VI','Lady Elizabeth Bowes-Lyon']" {
   "'George VI'" [
     fillcolor = "white:green"
     label = "George VI"
   ]
   "'Lady Elizabeth Bowes-Lyon'" [
     fillcolor = "white:red"
     label = "Lady Elizabeth Bowes-Lyon"
   ]
  }
  "'Rose Bowes-Lyon'" [
    fillcolor = "white:red"
    label = "Rose Bowes-Lyon"
  ]
 }
 subgraph "3" {
   rank = "same"
   rankdir = "LR"
  "'Elizabeth II'" [
    fillcolor = "white:red"
    label = "Elizabeth II"
  ]
 }
 "['George V','Princess Mary of Teck']" [
   shape = "point"
   width = "0.1"
   height = "0.1"
 ]
 "['Cecilia Cavendish-Bentinck','Claude Bowes-Lyon']" [
   shape = "point"
   width = "0.1"
   height = "0.1"
 ]
 "['George VI','Lady Elizabeth Bowes-Lyon']" [
   shape = "point"
   width = "0.1"
   height = "0.1"
 ]
 "'George V'" -> "['George V','Princess Mary of Teck']" [
 ]
 "'Princess Mary of Teck'" -> "['George V','Princess Mary of Teck']" [
 ]
 "['George V','Princess Mary of Teck']" -> "'George VI'" [
 ]
 "'Cecilia Cavendish-Bentinck'" -> "['Cecilia Cavendish-Bentinck','Claude Bowes-Lyon']" [
 ]
 "'Claude Bowes-Lyon'" -> "['Cecilia Cavendish-Bentinck','Claude Bowes-Lyon']" [
 ]
 "['Cecilia Cavendish-Bentinck','Claude Bowes-Lyon']" -> "'Lady Elizabeth Bowes-Lyon'" [
 ]
 "['Cecilia Cavendish-Bentinck','Claude Bowes-Lyon']" -> "'Rose Bowes-Lyon'" [
 ]
 "'George VI'" -> "['George VI','Lady Elizabeth Bowes-Lyon']" [
 ]
 "'Lady Elizabeth Bowes-Lyon'" -> "['George VI','Lady Elizabeth Bowes-Lyon']" [
 ]
 "['George VI','Lady Elizabeth Bowes-Lyon']" -> "'Elizabeth II'" [
 ]
}

我的要求是让气流从开始日期到当前日期执行我的任务。看起来它可以运行的骰子数量有限制，但我无法弄清楚这一点。请建议。

我的气流版本是1.7.1.2

Answer 1

不确定我是否帮助您，您是否希望从某个日期开始每天的工作直到接下来的16天？如果是这样，它与 airflow.cfg 没有任何关系，你发布的设置实际上可以“杀死”你的机器以防你有很多Dags，这是关于任务实例将并行运行并安排。

我想您需要的是定义 start_date 和 end_date 。您可以在构建将保留作业的DAG时执行此操作。一种方法是定义default_args，然后在你的dag上使用它们。

default_args = {
     'owner': 'userXX',
     'start_date': datetime(2014, 05, 26),
     'end_date': datetime(2014, 06, 11),
     'email': ['mail1..','mail2...'],
    ...
}

mydag = DAG(
           PARENT_ID, default_args=default_args,
           schedule_interval="@daily",
           user_defined_macros=user_defined_macros
)

_some_Operator(
                task_id='task_idxx',
                .. other settings/jobrun...
                dag=mydag,

            )

希望这有帮助

气流回填不起作用

1 个答案: