Question

我正在使用apache气流来运行我的dags。

我遇到一个例外：


*** Log file does not exist: /opt/airflow/logs/download2/download2/2020-07-26T15:00:00+00:00/1.log
*** Fetching from: http://fb3393f5f01e:8793/log/download2/download2/2020-07-26T15:00:00+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='fb3393f5f01e', port=8793): Max retries exceeded with url: /log/download2/download2/2020-07-26T15:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ba66d7b70>: Failed to establish a new connection: [Errno 111] Connection refused',))

我的用于Web服务器，调度程序和Postgres的docker compose文件是：

version: "2.1"
services:
  postgres_airflow:
    image: postgres:12
    environment:
        - POSTGRES_USER=airflow
        - POSTGRES_PASSWORD=airflow
        - POSTGRES_DB=airflow
    ports:
        - "5432:5432"

  postgres_Service:
    image: postgres:12
    environment:
        - POSTGRES_USER=developer
        - POSTGRES_PASSWORD=secret
        - POSTGRES_DB=service_db
    ports:
        - "5433:5432"
 
  scheduler:
    image: apache/airflow
    restart: always
    depends_on:
      - postgres_airflow
      - postgres_Service
      - webserver
    env_file:
      - .env
    volumes:
        - ./dags:/opt/airflow/dags
    command: scheduler
    healthcheck:
        test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
        interval: 30s
        timeout: 30s
        retries: 3

  webserver:
    image: apache/airflow
    restart: always
    depends_on:
        - pg_airflow
        - pg_metadata
        - tenants-registry-api
        - metadata-api
    env_file:
      - .env
    volumes:
        - ./dags:/opt/airflow/dags
        - ./scripts:/opt/airflow/scripts
    ports:
        - "8080:8080"
    entrypoint: ./scripts/airflow-entrypoint.sh
    healthcheck:
        test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
        interval: 30s
        timeout: 30s
        retries: 3

使用PythonVirtualenvOperator时出现此异常。

我的dag文件是：

from datetime import datetime

from airflow import DAG

from airflow.operators.python_operator import PythonOperator

default_args = {'owner': 'airflow',
                'start_date': datetime(2018, 1, 1)
                }

dag = DAG('download2',
          schedule_interval='0 * * * *',
          default_args=default_args,
          catchup=False)


def hello_world_py():
    return "data"


with dag:
    t1 = PythonOperator(
        task_id='download2',
        python_callable=hello_world_py,
        op_kwargs=None,
        provide_context=True,
        dag=dag
    )

env文件：

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql://airflow:airflow@postgres_airflow:5432/airflow
AIRFLOW__CORE__FERNET_KEY=XXXX
AIRFLOW_CONN_METADATA_DB=postgres://developer:secret@postgres_Service:5432/service_db
AIRFLOW__VAR__METADATA_DB_SCHEMA=service_db
AIRFLOW__WEBSERVER__BASE_URL=http://0.0.0.0:8080/

我还明确设置了AIRFLOW__CORE__REMOTE_LOGGING = False以禁用远程日志，但仍然会出现异常。还尝试将所有内容放入网桥网络中。尽管DAG通过了，但对我没有任何帮助。

还尝试添加：

    image: apache/airflow
    restart: always
    depends_on:
      - scheduler
    volumes:
      - ./dags:/opt/airflow/dags
    env_file:
      - .env
    ports:
    - 8793:8793
    command: worker

对我不起作用

Answer 1

您需要在docker-compose中公开工作日志服务器端口（默认情况下，airflow.cfg中的worker_log_server_port设置为8793），例如：

worker:
  image: apache/airflow
  ...
  ports:
    - 8793:8793

Answer 2

这是一种稍微不同的方法，我见过一些人在他们自己的容器中运行 scheduler 和 webserver 并使用 LocalExecutor（我猜这是案例在这里）：

将主机日志目录作为卷安装到 scheduler 和 webserver 容器中：

    volumes:
        - /location/on/host/airflow/logs:/opt/airflow/logs

确保气流容器内的用户（通常为 airflow）具有读取和写入该目录的权限。如果权限错误，您会看到类似帖子中的错误。

不过，这可能不会超过 LocalExecutor 的使用。

气流日志文件异常

2 个答案: