Question

我们正在使用芹菜和烧瓶应用。我们的各种任务有很多队列，我们使用supervisord来运行队列。我们使用cloudamqp 代理。

示例管理程序配置如下所示：

[program:my-queue]
command=/home/ubuntu/opt/proect/venv/bin/celery -A async_runner worker -Q my_queue --loglevel=INFO --without-gossip --without-mingle --autoscale=1,1  -c 1
environment=PYTHONPATH=/home/ubuntu/opt/project/,PRODUCTION_ENVIRONMENT=true
directory=/home/ubuntu/opt/project/app

process_name = %(program_name)s_%(process_num)02d
user=ubuntu
numprocs=2
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
priority=998

我们得到以下错误，队列就停止了。

[2017-04-06 12:43:06,759: WARNING/MainProcess] /home/ubuntu/opt/project/venv/local/lib/python2.7/site-packages/kombu/pidbox.py:75: UserWarning: A node named celery@ip-xxx-yy-yy-yy is already using this process mailbox!

Maybe you forgot to shutdown the other node or did not do so properly?
Or if you meant to start multiple nodes on the same host please make sure
you give each node a unique node name!

  warnings.warn(W_PIDBOX_IN_USE.format(node=self))

问题：如何为每个节点命名？

当我运行celery -A async_runner status时，它会在线提供5个节点消息。

celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK

5 nodes online.

Answer 1

以下是我们为公司的每个队列运行一名芹菜工人的方法。入口点包含：

echo QUEUES: ${QUEUES}  # comma separated list of queue names
export NUM_QUEUES=$(python -c "print len('$QUEUES'.split(','))")
echo NUM_QUEUES: ${NUM_QUEUES}

supervisord &

主管配置文件包含：

[program:worker]
command=/path/to/celery_worker.sh %(process_num)s
# supervisor uses the special ENV_ prefix to get the environment variable set above
numprocs=%(ENV_NUM_QUEUES)s

最后celery_worker.sh包含类似的内容：

QUEUE=$(python -c "print '$QUEUES'.split(',')[$1]")  # $1 = process_num above
celery worker -n "worker.${QUEUE}" -Q ${QUEUE}  # -n sets the name of the worker

Answer 2

使用选项-n SOMENODENAME@%%h。 @%%h将确保每个工作进程都有唯一的节点名

如何为芹菜中的每个节点命名

2 个答案: