Question

我遵循tutorial关于插件的信息。

但是最重要的答案也帮不了我。

这是我项目的简化版本：

airflow_home
├── dags
│   └── etl.py  
└── plugins
    ├── __init__.py
    └── operators
        ├── __init__.py
        └── dump_file.py

plugins / operators / dump_file.py

from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults

class DumpCsvFileToPostgres(BaseOperator):
    [...]

# plugins/__init__.py

from airflow.plugins_manager import AirflowPlugin
from plugins.operators.dump_file import DumpCsvFileToPostgres

# Defining the plugin class
class CustomPlugin(AirflowPlugin):
    name = "custom_plugin"
    operators = [
        DumpCsvFileToPostgres
    ]
    helpers = []

dags/etl.py

# tried 
# from airflow.operators import DumpCsvFileToPostgres
# from airflow.operators.custom_plugin import DumpCsvFileToPostgres
from custom_plugin import DumpCsvFileToPostgres

[...]

我仍然得到

损坏的DAG：[/root/airflow/dags/etl_dag.py]没有名为'custom_plugin'的模块

我的网络服务器和调度程序正在使用docker-compose运行

version: '3.7'
services:
  [...]

  webserver:
    image: godatadriven/airflow:latest
    environment:
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER:-airflow}:${POSTGRES_PASSWORD:-airflow}@postgres:5432/${POSTGRES_DB:-airflow}
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - WAIT_FOR=postgres:5432
    depends_on:
      - postgres
    volumes:
      - ./dags:/root/airflow/dags
      - ./plugins:/root/airflow/plugins
      - ./logs:/root/airflow/logs
      - ./environment.yml:/dependencies/environment.yml
    ports:
      - "8080:8080"
    command: upgradedb_webserver

  scheduler:
    image: godatadriven/airflow:latest
    environment:
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER:-airflow}:${POSTGRES_PASSWORD:-airflow}@postgres:5432/${POSTGRES_DB:-airflow}
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - WAIT_FOR=webserver:8080
    depends_on:
      - webserver
    volumes:
      - ./dags:/root/airflow/dags
      - ./plugins:/root/airflow/plugins
      - ./logs:/root/airflow/logs
      - ./environment.yml:/dependencies/environment.yml
      - ./dataset/Iowa_Liquor_Sales.csv:/root/airflow/dataset/dataset.csv
    command: scheduler

Answer 1

您的etl.py中的import语句应为from airflow.macros import custom_plugin

还要检查plugins_folder中airflow.cfg键的值是否指向plugins目录。

Answer 2

在许多解决方案中，一种可能的解决方案是：

在dags/etl.py中，更改此行

from custom_plugin import DumpCsvFileToPostgres

到

from plugins.operators.dump_file import DumpCsvFileToPostgres

Answer 3

嘿，您看到堆栈溢出了吗？看来，导入自定义运算符的方法要简单得多

https://dotnetfiddle.net/Kxpagh

气流-无法导入自定义插件

3 个答案: