与Celery执行程序一起将Airflow作为集群运行时的问题

时间:2019-01-25 14:43:58

标签: rabbitmq celery airflow flower

我正在尝试将Airflow设置为群集。 Celery执行器在执行作业时无法连接到RabbitMQ 配置如下:

Machine 1: webserver and scheduler
Machine 2: webserver
Machine 3: worker
Machine 4: Rabbitmq

Airflow version: v1.8.0
Celery version: 4.1.0
Flower UI version: 0.9.1



**airflow.cfg**
airflow_home = ~/airflow
dags_folder = ~/airflow/dags
base_log_folder = ~/airflow/logs
executor = CeleryExecutor
sql_alchemy_conn = mysql://reco_airflow:password@10.32.170.111:3306/airflow
sql_alchemy_pool_size = 5
dag_concurrency = 16
dags_are_paused_at_creation = False
plugins_folder = ~/airflow/plugins
# Secret key to save connection passwords in the db
api_client = airflow.api.client.local_client
endpoint_url = http://10.34.110.227:8080
base_url = http://10.34.110.227:8080
web_server_host = 0.0.0.0
web_server_port = 8080
workers = 4
broker_url = amqp://guest:guest@10.34.94.212:5672//
celery_result_backend = db+mysql://reco_airflow:password@10.32.170.109:3306/airflow
flower_host = 10.34.110.227
flower_port = 5555
default_queue = queue

使用celery执行程序成功运行Dag。但是我看不到与Rabbitmq的任何连接(它总是空闲的)。 另外,如果我尝试使用flower,则UI会打开,但会继续加载并在约5秒后停止响应。日志中没有错误。我在配置中缺少什么吗?

0 个答案:

没有答案