Question

在airflow.conf的kubernetes部分下，可以为dag和log安装volumeMount，例如

[kubernetes]
airflow_configmap = airflow_config
worker_container_repository = airflow
worker_container_tag = runner2
worker_container_image_pull_policy = IfNotPresent
delete_worker_pods = true
dags_volume_claim = airflow-dags-pvc
dags_volume_subpath = airflow/development/dags
logs_volume_claim = airflow-logs-pvc
logs_volume_subpath = airflow/development/logs
namespace = development

这按预期工作。意味着我可以看到工作人员吊舱已成功将这两个卷及其相关的volumeMount安装在容器中。

   "volumeMounts": [
      {
        "name": "airflow-dags",
        "readOnly": true,
        "mountPath": "/usr/local/airflow/dags",
        "subPath": "airflow/development/dags"
      },
      {
        "name": "airflow-logs",
        "mountPath": "/usr/local/airflow/logs",
        "subPath": "airflow/development/logs"
      },

但是，我的工作人员吊舱具有从目录airflow/development/plugins和airflow/development/libs拾取自定义气流插件的依赖性。因此，我需要使用来自NFS服务器的相关子路径将更多的volumeMount添加到工作容器中。我该如何实现？我尝试搜索任何相关的配置值，但找不到任何。

更新：我将executor_config作为executor_config={"KubernetesExecutor": {"image": "airflow:runner2"}}传递到dags传感器任务之一。通过查看代码airflow/contrib/kubernetes/worker_configuration.py，似乎如果我将volume和volumeMounts作为此配置的一部分传递，它应该可以工作。我会尝试这个并在这里更新。

Answer 1

对不起，我想回答我自己的问题。也许会帮助某人。在dag文件中，我在其中定义任务，只需要按如下所示添加executor_config；

        IngestionStatusSensor(
            task_id=...,
            executor_config={"KubernetesExecutor": {
                                  "image": "airflow:runner2",
                                  "volume_mounts": [
                                      {
                                          "name": "airflow-dags",
                                          "mountPath": "/usr/local/airflow/libs",
                                          "subPath": "airflow/development/libs"
                                      },
                                      {
                                          "name": "airflow-dags",
                                          "mountPath": "/usr/local/airflow/plugins",
                                          "subPath": "airflow/development/plugins"
                                      }],
                                  }
                             },
            dag=dag,
            ingestion_feed=table,
            poke_interval=60 * 30,
            offset=0,
            start_date=start_date
        )

其中airflow-dags卷已经由Pod创建者定义，该创建者声明了在airflow.conf的kubernetes部分下的配置中定义的PVC，例如dags_volume_claim = airflow-pvc。

在Airflow Worker的pod中添加自定义volumeMount（使用k8s Executor）

1 个答案: