supervisord和芹菜暖关机

时间:2014-05-17 18:09:49

标签: ubuntu redis celery supervisord supervisor

每当我跑" sudo shutdown -h now"在一个运行Supervisor和Celery的ubuntu实例上,似乎Celery只是在处理任务的过程中关闭,当看到尾部-f我没有看到"热关机"在Celery在前台运行时键入ctrl-C时通常会看到的消息。

这对我来说是个大问题,因为服务器可能会随时终止,因此等待Celery完成其任务的处理至关重要。可能需要几秒钟,10分钟或10个小时,但关闭信号应始终让Celery完成其工作。 我在我的celeryd配置文件中添加了stopasgroup和killasgroup,看起来好像从主管停止Celery会停止所有子进程,但不知何故它似乎忽略了" sudo shutdown -h now"

有没有人知道为什么Celery没有显示通常的" Warm Shutdown"它会正常吗?

这是一个工人conf的例子:

[program:celery-worker]
command=/path/to/script-running-celery.sh
directory=/path/to/celery/project/tasks
user=myuser
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 60
killasgroup=true
stopasgroup=true
priority=50

这是正在运行的实际shell脚本:

#!/bin/bash

export PATH="$PATH:/mylocal-envs/localenv";
cd /path/to/celery/project/tasks;
exec /mylocal-envs/localenv/bin/celery worker -c8 -n my_worker_%h_$(uuid) --without-gossip -A tasks --queues=myqueue --loglevel=INFO -Ofair

你能看出为什么关闭命令不会让Celery在暖关机中运行并结束当前任务的原因吗?

编辑:只是注意,当直接运行我的脚本而不使用Supervisor并从另一个ssh窗口关闭时,我仍然得到相同的行为:工作人员正在处理任务,然后收到关闭信号并关闭ssh会话。以下是运行脚本时工作人员前台发生的情况:

[2014-05-17 19:29:02,955: INFO/MainProcess] Task tasks.a_task[5143a89d-e077-4145-ba94-0932d0f81d9e] succeeded in 15.523131618s: u''
[2014-05-17 19:29:03,621: INFO/MainProcess] Received task: tasks.a_task[b61a30ba-2fbb-475a-9cb2-ec88d20a8885]

Broadcast message from jdoe@10.0.0.12
    (/dev/pts/1) at 19:29 ...

The system is going down for halt NOW!
[2014-05-17 19:29:06,846: INFO/MainProcess] Received task: tasks.a_task[9146f9f4-f06c-4563-b8d1-43c8d9367d9b]
[2014-05-17 19:29:08,089: INFO/MainProcess] Task tasks.a_task[b4c7dacc-3f6d-42e5-ab67-f33748f0af14] succeeded in 16.49714411s: u''
Connection to xx.xx.xxx.xx closed by remote host.
Connection to xx.xx.xxx.xx closed.

您仍然可以看到任务成功,收到另一个任务。看着花,我看到一堆已经开始的任务,现在已经停留在"开始"或"收到"。

编辑:尝试直接在command =选项中运行celery而不是运行shell脚本后,我注意到了相同的行为。如果我在前台的ssh上运行我的脚本(在调试模式下使用celery),我在点击ctrl-C时会得到以下内容:

worker: Hitting Ctrl+C again will terminate all running tasks!
289
290worker: Warm shutdown (MainProcess)
291[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Hub...
292[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Pool...
293[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Consumer...
294[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Worker: Stopping Consumer...
295[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Connection...
296[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Events...
297[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Mingle...
298[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Tasks...
299[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Control...
300[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Heart...
301[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing event loop...
302[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping event loop...
303[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping Heart...
304[2014-05-18 13:12:49,564: DEBUG/MainProcess] | Consumer: Stopping Control...
305[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Tasks...
306[2014-05-18 13:12:49,570: DEBUG/MainProcess] Cancelling task consumer...
307[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Mingle...
308[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Events...
309[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Consumer: Stopping Connection...
310[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Worker: Stopping Pool...
311[2014-05-18 13:12:50,608: DEBUG/MainProcess] | Worker: Stopping Hub...
312[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Heart...
313[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Control...
314[2014-05-18 13:12:50,613: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
315[2014-05-18 13:12:50,614: DEBUG/MainProcess] Cancelling task consumer...
316[2014-05-18 13:12:50,614: DEBUG/MainProcess] Closing consumer channel...
317[2014-05-18 13:12:50,615: DEBUG/MainProcess] | Consumer: Shutdown Events...
318[2014-05-18 13:12:50,616: DEBUG/MainProcess] | Consumer: Shutdown Connection...
319[2014-05-18 13:12:50,617: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

这看起来像我希望与主管一起发生的正常关机过程。 另一方面,停止使用主管给我以下内容:

[2014-05-18 13:14:28,369: DEBUG/MainProcess] pidbox received method ping() [reply_to:{u'routing_key': u'1bcaf3a6-0c77-33b9-9d46-56e8634e5b1d', u'exchange': u'reply.celery.pidbox'} ticket:161fcbdb-dac0-48c9-ac13-ab7a75bfdc85]

没有像你在点击ctrl-c时看到的关闭跟踪。它只是停止了。

1 个答案:

答案 0 :(得分:0)

这似乎是Ubuntu(3.0a)附带的Supervisor版本中的一个错误。切换到当前稳定版Supervisor(手动安装在/ usr / local / bin中)后,我可以看到日志确实收到了sigterm。