Question

我有一个部分工作的Python Celery应用程序，它能够在远程工作人员上执行作业。但是，同步始终无法正常工作。

花了一些时间来追查问题，我意识到这取决于工人没有回来。他们能够完成任务，但有时他们只是在返回时冻结。虽然通常我可以看到Task ... succeeded in $time消息，但在这种情况下消息不会显示。

如果我尝试发送SIGQUIT（仅使用 Ctrl + C ），工作人员将继续被冻结。我必须用SIGTERM杀死它。

任何提示？

旁注

没有任务调用任务。每个任务都由主进程调用

工作流程如下

async = bar.s(...) | baz.s(...)
# later on...
async.parent.get()
# later on...
async.get()

...但即使在不使用链条的情况下执行bar和baz，效果也会持续存在。设置不会忽略结果，并且ignore_result=True 不指定在任何位置。事实上，该系统最初工作正常，然后卡住了。

celery inspect active命令在被阻止的工作程序上返回作业规范本身。

当前配置

app = celery.Celery(
    broker='amqp://...'
    backend='amqp'
)

app.conf.update(
    CELERY_TASK_SERIALIZER='json',
    CELERY_RESULT_SERIALIZER='json',
    CELERY_ACCEPT_CONTENT=['json'],
    CELERY_TIMEZONE = 'UTC',
    CELERY_ENABLE_UTC = True,
    CELERY_RESULT_PERSISTENT = True,
    CELERY_TRACK_STARTED = True
)

使用RabbitMQ的Redis insead也会发生同样的情况。

从工人登录：

有时可能会发生 billiard 的 Broken Pipe 通知，但这似乎没有关系，因为它有时只会发生。

[2014-11-25 10:47:20,995: DEBUG/MainProcess] | Worker: Starting Hub
[2014-11-25 10:47:20,996: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:20,996: DEBUG/MainProcess] | Worker: Starting Pool
[2014-11-25 10:47:21,002: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:21,004: DEBUG/MainProcess] | Worker: Starting Consumer
[2014-11-25 10:47:21,004: DEBUG/MainProcess] | Consumer: Starting Connection
[2014-11-25 10:47:22,053: INFO/MainProcess] Connected to redis://:**@...
[2014-11-25 10:47:22,053: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:22,053: DEBUG/MainProcess] | Consumer: Starting Events
[2014-11-25 10:47:22,094: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:22,094: DEBUG/MainProcess] | Consumer: Starting Mingle
[2014-11-25 10:47:22,095: INFO/MainProcess] mingle: searching for neighbors
[2014-11-25 10:47:23,152: INFO/MainProcess] mingle: all alone
[2014-11-25 10:47:23,152: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:23,152: DEBUG/MainProcess] | Consumer: Starting Gossip
[2014-11-25 10:47:23,183: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:23,184: DEBUG/MainProcess] | Consumer: Starting Tasks
[2014-11-25 10:47:23,193: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:23,194: DEBUG/MainProcess] | Consumer: Starting Control
[2014-11-25 10:47:23,223: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:23,223: DEBUG/MainProcess] | Consumer: Starting Heart
[2014-11-25 10:47:23,225: DEBUG/MainProcess] ^-- substep ok
[2014-11-25 10:47:23,225: DEBUG/MainProcess] | Consumer: Starting event loop
[2014-11-25 10:47:23,226: INFO/MainProcess] celery@... ready.
[2014-11-25 10:47:23,226: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2014-11-25 10:47:23,227: DEBUG/MainProcess] basic.qos: prefetch_count->4
[2014-11-25 10:47:24,095: INFO/MainProcess] Received task: simulation.launch[76b59c23-9035-49b9-9997-fa9d6c347a76]
[2014-11-25 10:47:24,096: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x26e6230> (args:(<all my args...
[2014-11-25 10:47:24,099: INFO/MainProcess] Received task: simulation.launch[13844d7b-90ad-427a-b7c0-4e5f4fd3aef0]
[2014-11-25 10:47:24,100: DEBUG/MainProcess] Task accepted: simulation.launch[76b59c23-9035-49b9-9997-fa9d6c347a76] pid:13997
[2014-11-25 10:47:24,102: INFO/MainProcess] Received task: simulation.launch[0e3a13ac-845e-47ab-b969-88931a7a4397]

<follows output of the task being executed. Then the task returns>

<silence for a long time>

[2014-11-25 10:54:32,244: ERROR/Worker-1] Pool process <Worker(Worker-1, started daemon)> error: OSError(32, 'Broken pipe')
Traceback (most recent call last):
  File "/home/testing/.local/lib/python2.7/site-packages/billiard/pool.py", line 289, in run
    sys.exit(self.workloop(pid=pid))
  File "/home/testing/.local/lib/python2.7/site-packages/billiard/pool.py", line 373, in workloop
    put((READY, (job, i, (False, einfo), inqW_fd)))
OSError: [Errno 32] Broken pipe

任务冻结返回

旁注

当前配置

从工人登录：

0 个答案: