限制Celery工作人员排队的任务编号

时间:2017-02-21 09:31:02

标签: python celery

我有一个最简单的Celery任务示例(case.py):

import time
import celery


app = celery.Celery('case', broker='redis://localhost')

@app.task
def do_sth():
    time.sleep(5)

我创建了一些任务实例:

>>> import case
>>> tasks_list = [case.do_sth.delay() for i in range(4)]

然后我希望拥有--concurrency=1参数的动态工作人数。这对我的情况至关重要,因为每个工作者只有一个并行任务,并且当某些任务在服务器上排队时添加工作者。如果某些任务没有其工作人员并且添加了新工作程序,则新工作人员应该处理排队(待处理但未执行)任务。但是当我打电话给$ celery -A case worker --loglevel=info --concurrency=1时,我有以下日志(不包括Celery徽标):

[tasks]
  . case.do_sth

[2017-02-21 10:03:18,454: INFO/MainProcess] Connected to redis://localhost:6379//
[2017-02-21 10:03:18,459: INFO/MainProcess] mingle: searching for neighbors
[2017-02-21 10:03:19,471: INFO/MainProcess] mingle: all alone
[2017-02-21 10:03:19,480: INFO/MainProcess] celery@pt0 ready.
[2017-02-21 10:03:19,658: INFO/MainProcess] Received task: case.do_sth[0c7b0f8c-d1f8-4cd8-a100-21ef6654e04c]
[2017-02-21 10:03:19,660: INFO/MainProcess] Received task: case.do_sth[f97ad614-017b-4a6c-90df-89dbed63e39b]
[2017-02-21 10:03:19,662: INFO/MainProcess] Received task: case.do_sth[b0166022-196f-451b-bcb6-78cdf0558803]
[2017-02-21 10:03:19,664: INFO/MainProcess] Received task: case.do_sth[b097e191-5bc4-44d9-bdcd-8aa74501e95d]
[2017-02-21 10:03:24,667: INFO/PoolWorker-1] Task case.do_sth[0c7b0f8c-d1f8-4cd8-a100-21ef6654e04c] succeeded in 5.006301835s: None
[2017-02-21 10:03:29,675: INFO/PoolWorker-1] Task case.do_sth[f97ad614-017b-4a6c-90df-89dbed63e39b] succeeded in 5.005384011s: None
[2017-02-21 10:03:34,683: INFO/PoolWorker-1] Task case.do_sth[b0166022-196f-451b-bcb6-78cdf0558803] succeeded in 5.005373027s: None
[2017-02-21 10:03:39,690: INFO/PoolWorker-1] Task case.do_sth[b097e191-5bc4-44d9-bdcd-8aa74501e95d] succeeded in 5.00531687s: None

与此同时,我开始了另一个工人(使用相同的命令),但它没有做任何事情:

[tasks]
  . case.do_sth

[2017-02-21 10:03:20,321: INFO/MainProcess] Connected to redis://localhost:6379//
[2017-02-21 10:03:20,326: INFO/MainProcess] mingle: searching for neighbors
[2017-02-21 10:03:21,339: INFO/MainProcess] mingle: all alone
[2017-02-21 10:03:21,352: INFO/MainProcess] celery@pt0 ready.

如果你查看日期,你可以看到第二个工人在第一个工人被触发后不到五秒钟。

是否有办法(某些工人或芹菜选项)将单个工作人员排队的任务限制为一个任务?

1 个答案:

答案 0 :(得分:1)

我认为你要找的东西叫做prefetch limit。它有效地减少了工人在给定时间可以预留的任务数量。

  

预取限制是工作人员可以为自己保留的任务(消息)数量的限制