我已经被这个问题困扰了一段时间了。
我按照步骤进行操作,成功完成了芹菜的任务,当我独自运行芹菜时,它的工作如预期的那样:
-------------- celery@xxxxcxxxxx.zzzz.org v4.3.0 (rhubarb)
---- **** -----
--- * *** * -- Linux-2.6.32-573.22.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago 2019-05-11 11:34:25
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: tasks:0x7fcb942e93d0
- ** ---------- .> transport: amqp://guest:**@xx.xx.xx.xx:5672//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 5 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. tasks.add
. tasks.rehi
但是,当我使用以下命令在airflow.cfg中进行配置时:Checkthis
这是我的airflow.cfg:
executor=CeleryExecutor
CELERY_BROKER_URL = 'amqp://guest:guest@xx.xx.xx:5672/'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/'
# The SqlAlchemy connection string to the metadata database.
# SqlAlchemy supports many different database engine, more information
# their website
#sql_alchemy_conn = sqlite:////home/apservices/airflow/airflow.db
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@yy.yy.yy.yy:5432/airflow
但是,当我运行airflow worker
时,只是没有选择airflow.cfg中给出的配置:
-------------- celery@xxx.xx.org v4.3.0 (rhubarb)
---- **** -----
--- * *** * -- Linux-2.6.32-573.22.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago 2019-05-11 12:00:57
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7fa16cf99750
- ** ---------- .> transport: sqla+mysql://airflow:airflow@localhost:3306/airflow
- ** ---------- .> results: mysql://airflow:**@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
完成Stacktrace:
[2019-05-11 12:00:56,936] {configuration.py:206}警告-部分/密钥 在配置[2019-05-11]中找不到[celery / celery_ssl_active] 12:00:56,936] {default_celery.py:41}警告-Celery执行器将 在没有SSL的情况下运行[2019-05-11 12:00:56,938] { init .py:45}信息- 使用执行器CeleryExecutor
-------------- celery@xx.xx.org v4.3.0(大黄) ---- **** ----- --- * * *-Linux-2.6.32-573.22.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago 2019-05-11 12:00:57 -*-**** --- -** ---------- [config] -** ----------。>应用:airflow.executors.celery_executor:0x7fa16cf99750 -** ----------。>运输:sqla + mysql:// airflow:airflow @ localhost:3306 / airflow -** ----------。>结果:mysql:// airflow: @localhost:3306 / airflow -*** --- * ---。>并发性:16(prefork) -******* ----->任务事件:OFF(启用-E监视此工作程序中的任务) --- ***** ----- -------------- [队列] 。>默认交换=默认(直接)键=默认
[2019-05-11 12:00:57,160:CRITICAL / MainProcess]无法恢复的错误: TypeError(“无效的参数'visibility_timeout'发送到 create_engine(),使用配置 MySQLDialect_mysqldb / QueuePool / Engine。请检查关键字 参数适用于组件的这种组合。”,) 追溯(最近一次通话):文件 “ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/worker/worker.py”, 205行,开始 self.blueprint.start(self)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/bootsteps.py”, 第119行,开始时 step.start(父)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/bootsteps.py”, 第369行,开始时 在开始时返回self.obj.start()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/worker/consumer/consumer.py”,第318行 blueprint.start(self)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/bootsteps.py”, 第119行,开始时 step.start(父)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/worker/consumer/tasks.py”, 第41行,开始时 c.connection,on_decode_error = c.on_decode_error,文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/celery/app/amqp.py”, TaskConsumer中的第297行 ** kw文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/messaging.py”, 第386行,在 init 中 self.revive(self.channel)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/messaging.py”, 复兴的第408行 self.declare()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/messaging.py”, 第421行,在声明中 queue.declare()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/entity.py”, 第608行,在声明中 self._create_queue(nowait = nowait,channel = channel)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/entity.py”, _create_queue中的第617行 self.queue_declare(nowait = nowait,被动= False,通道=通道)文件 “ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/entity.py”, 第652行,在queue_declare中 nowait = nowait,文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/virtual/base.py”, 在queue_declare中的第531行 self._new_queue(queue,** kwargs)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/ init 。 py”, _new_queue中的第82行 self._get_or_create(queue)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init.py”, 第70行,在_get_or_create中 obj = self.session.query(self.queue_cls)\文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init .py”, 会话中的第65行 _,会话= self._open()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init.py ”, _open中的第56行 engine = self._engine_from_config()文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/kombu/transport/sqlalchemy/init.py”, _engine_from_config中的第51行 返回create_engine(conninfo.hostname,** transport_options)文件 “ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/sqlalchemy/engine/init.py”, create_engine中的386行 返回strategy.create(* args,** kwargs)文件“ /opt/cloudera/parcels/Anaconda-2.5.0/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py”, 第144行,在创建中 engineclass。 name ))TypeError:使用配置发送给create_engine()的无效参数'visibility_timeout' MySQLDialect_mysqldb / QueuePool / Engine。请检查关键字 参数适用于组件的这种组合。 [2019-05-11 12:00:57,898] {configuration.py:206}警告-部分/密钥 在配置[2019-05-11]中找不到[celery / celery_ssl_active] 12:00:57,898] {default_celery.py:41}警告-Celery执行器将 在没有SSL的情况下运行[2019-05-11 12:00:57,900] { init .py:45}信息- 使用执行程序CeleryExecutor启动烧瓶[2019-05-11 12:00:58,083] {_internal.py:122}信息-*在http://0.0.0.0:8793/上运行(按 CTRL + C退出)
到目前为止,我不担心TypeError: Invalid argument(s) 'visibility_timeout'
,因为我确信在airflow.conf中传递的配置即将被使用。
让我知道我在想什么,我们将不胜感激!
干杯!