Airflow Celery和Redis作业处于等待状态

时间:2018-09-26 10:18:47

标签: redis celery airflow airflow-scheduler

我试图以芹菜为原料,以redis作为经纪人,但工作陷入等待状态。 气流在本地运行,我正在使用示例DAG进行测试。

executor = CeleryExecutor


sql_alchemy_conn = mysql://root@localhost/airflow


sql_alchemy_pool_size = 5



[celery]
# This section only applies if you are using the CeleryExecutor in
# [core] section above

# The app name that will be used by celery
celery_app_name = airflow.executors.celery_executor

# The concurrency that will be used when starting workers with the
# "airflow worker" command. This defines the number of task instances that
# a worker will take, so size up your workers based on the resources on
# your worker box and the nature of your tasks
celeryd_concurrency = 16

# When you start an airflow worker, airflow starts a tiny web server
# subprocess to serve the workers local log files to the airflow main
# web server, who then builds pages and sends them to users. This defines
# the port on which the logs are served. It needs to be unused, and open
# visible from the main web server to connect into the workers.
worker_log_server_port = 8793

# The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally
# a sqlalchemy database. Refer to the Celery documentation for more
# information.
broker_url = redis://localhost:6379/0

# Another key Celery setting
celery_result_backend = redis://localhost:6379/0

# Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start
# it `airflow flower`. This defines the port that Celery Flower runs on
flower_port = 5555

# Default queue that tasks get assigned to and that worker listen on.
default_queue = default

0 个答案:

没有答案