CeleryExecutor的气流未从airflow.cfg

时间:2019-05-11 16:08:29

标签: python-2.7 celery airflow celery-task airflow-scheduler

我已经被这个问题困扰了一段时间了。

我按照步骤进行操作,成功完成了芹菜的任务,当我独自运行芹菜时,它的工作如预期的那样:

-------------- celery@xxxxcxxxxx.zzzz.org v4.3.0 (rhubarb)
---- **** -----
--- * ***  * -- Linux-2.6.32-573.22.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago 2019-05-11 11:34:25
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x7fcb942e93d0
- ** ---------- .> transport:   amqp://guest:**@xx.xx.xx.xx:5672//
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 5 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
[tasks]
  . tasks.add
  . tasks.rehi

但是,当我使用以下命令在airflow.cfg中进行配置时:Checkthis

这是我的airflow.cfg:

executor=CeleryExecutor

CELERY_BROKER_URL = 'amqp://guest:guest@xx.xx.xx:5672/'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/'

# The SqlAlchemy connection string to the metadata database.
# SqlAlchemy supports many different database engine, more information
# their website
#sql_alchemy_conn = sqlite:////home/apservices/airflow/airflow.db
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@yy.yy.yy.yy:5432/airflow

但是,当我运行airflow worker时,只是没有选择airflow.cfg中给出的配置:

-------------- celery@xxx.xx.org v4.3.0 (rhubarb)
---- **** -----
--- * ***  * -- Linux-2.6.32-573.22.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago 2019-05-11 12:00:57
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         airflow.executors.celery_executor:0x7fa16cf99750
- ** ---------- .> transport:   sqla+mysql://airflow:airflow@localhost:3306/airflow
- ** ---------- .> results:     mysql://airflow:**@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> default          exchange=default(direct) key=default

完成Stacktrace:

  

[2019-05-11 12:00:56,936] {configuration.py:206}警告-部分/密钥   在配置[2019-05-11]中找不到[celery / celery_ssl_active]   12:00:56,936] {default_celery.py:41}警告-Celery执行器将   在没有SSL的情况下运行[2019-05-11 12:00:56,938] { init .py:45}信息-   使用执行器CeleryExecutor

     

-------------- celery@xx.xx.org v4.3.0(大黄)   ---- **** -----   --- * * *-Linux-2.6.32-573.22.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago   2019-05-11 12:00:57   -*-**** ---   -** ---------- [config]   -** ----------。>应用:airflow.executors.celery_executor:0x7fa16cf99750   -** ----------。>运输:sqla + mysql:// airflow:airflow @ localhost:3306 / airflow   -** ----------。>结果:mysql:// airflow: @localhost:3306 / airflow   -*** --- * ---。>并发性:16(prefork)   -******* ----->任务事件:OFF(启用-E监视此工作程序中的任务)   --- ***** ----- -------------- [队列]                   。>默认交换=默认(直接)键=默认

     

[2019-05-11 12:00:57,160:CRITICAL / MainProcess]无法恢复的错误:   TypeError(“无效的参数'visibility_timeout'发送到   create_engine(),使用配置   MySQLDialect_mysqldb / QueuePool / Engine。请检查关键字   参数适用于组件的这种组合。”,)   追溯(最近一次通话):文件   “ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/worker/worker.py”,   205行,开始       self.blueprint.start(self)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/bootsteps.py”,   第119行,开始时       step.start(父)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/bootsteps.py”,   第369行,开始时       在开始时返回self.obj.start()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/worker/consumer/consumer.py”,第318行       blueprint.start(self)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/bootsteps.py”,   第119行,开始时       step.start(父)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/worker/consumer/tasks.py”,   第41行,开始时       c.connection,on_decode_error = c.on_decode_error,文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/app/amqp.py”,   TaskConsumer中的第297行       ** kw文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/messaging.py”,   第386行,在 init 中       self.revive(self.channel)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/messaging.py”,   复兴的第408行       self.declare()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/messaging.py”,   第421行,在声明中       queue.declare()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/entity.py”,   第608行,在声明中       self._create_queue(nowait = nowait,channel = channel)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/entity.py”,   _create_queue中的第617行       self.queue_declare(nowait = nowait,被动= False,通道=通道)文件   “ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/entity.py”,   第652行,在queue_declare中       nowait = nowait,文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/virtual/base.py”,   在queue_declare中的第531行       self._new_queue(queue,** kwargs)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/ init 。 py”,   _new_queue中的第82行       self._get_or_create(queue)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init.py”,   第70行,在_get_or_create中       obj = self.session.query(self.queue_cls)\文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init .py”,   会话中的第65行       _,会话= self._open()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init.py ”,   _open中的第56行       engine = self._engine_from_config()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init.py”,   _engine_from_config中的第51行       返回create_engine(conninfo.hostname,** transport_options)文件   “ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/sqlalchemy/engine/init.py”,   create_engine中的386行       返回strategy.create(* args,** kwargs)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py”,   第144行,在创建中       engineclass。 name ))TypeError:使用配置发送给create_engine()的无效参数'visibility_timeout'   MySQLDialect_mysqldb / QueuePool / Engine。请检查关键字   参数适用于组件的这种组合。   [2019-05-11 12:00:57,898] {configuration.py:206}警告-部分/密钥   在配置[2019-05-11]中找不到[celery / celery_ssl_active]   12:00:57,898] {default_celery.py:41}警告-Celery执行器将   在没有SSL的情况下运行[2019-05-11 12:00:57,900] { init .py:45}信息-   使用执行程序CeleryExecutor启动烧瓶[2019-05-11 12:00:58,083]   {_internal.py:122}信息-*在http://0.0.0.0:8793/上运行(按   CTRL + C退出)

到目前为止,我不担心TypeError: Invalid argument(s) 'visibility_timeout',因为我确信在airflow.conf中传递的配置即将被使用。

让我知道我在想什么,我们将不胜感激!

干杯!

0 个答案:

没有答案