Question

我是Airflow的新手，现在已经尝试了一段时间，使用KubernetesPodOperator类来运行任务没有任何运气。创建了pod并从git repo同步了DAG，我没有看到任何检索DAG的错误，但pod很快因以下错误而失败：

[2020-07-16 23:55:16,593] {__init__.py:51} INFO - Using executor LocalExecutor
[2020-07-16 23:55:16,594] {dagbag.py:396} INFO - Filling up the DagBag from /opt/airflow/dags/kubernetes_executor.py
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 75, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
    dag = get_dag(args)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/bin/cli.py", line 149, in get_dag
    'parse.'.format(args.dag_id))
airflow.exceptions.AirflowException: dag_id could not be found: kubernetes_sample. Either the dag did not exist or it failed to parse.

我正在使用Helm在EKS上部署Airflow 1.10.10。以下是我的气流配置文件。我注意到这是与Airflow- dag_id could not be found issue when using kubernetes executor相同的问题，我为AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT和AIRFLOW__KUBERNETES__GIT_SYNC_ROOT尝试了不同的组合，但是没有运气。

有什么想法吗？我完全被困住了。

airflow:
  executor: KubernetesExecutor
  config:
    AIRFLOW__CORE__EXECUTOR: KubernetesExecutor
    AIRFLOW__CORE__LOGGING_LEVEL: INFO
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://postgres:airflow@airflow-postgresql:5432/airflow"
    AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.10-python3.6"
    AIRFLOW__KUBERNETES__NAMESPACE: airflow
    AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: airflow
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: false
    AIRFLOW__KUBERNETES__RUN_AS_USER: 50000
    AIRFLOW__KUBERNETES__FS_GROUP: 65533
    AIRFLOW__KUBERNETES__GIT_REPO: ssh://git@github.com/rootstrap/champz-extractor.git
    AIRFLOW__KUBERNETES__GIT_BRANCH: seba
    AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: /opt/airflow/dags
    AIRFLOW__KUBERNETES__GIT_SYNC_DEST: ""
    AIRFLOW__KUBERNETES__GIT_SYNC_ROOT: /opt/airflow/dags  
    AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: airflow-secrets

workers:
  enabled: false
flower:
  enabled: false
redis:
  enabled: false

dags:
  
  git:
    url: ssh://git@github.com/rootstrap/champz-extractor.git
    repoHost: github.com
    repoPort: 22
    ref: seba
    secret: airflow-git-keys
    privateKeyName: id_rsa

    gitSync:
      enabled: true
      refreshTime: 60
  
  initContainer:
    enabled: true
      
  installRequirements: true


ingress:
  enabled: true

无法使用KubernetesExecutor运行Airflow DAG-AirflowException：找不到dag_id

0 个答案: