我有一个最简单的Celery任务示例(case.py
):
import time
import celery
app = celery.Celery('case', broker='redis://localhost')
@app.task
def do_sth():
time.sleep(5)
我创建了一些任务实例:
>>> import case
>>> tasks_list = [case.do_sth.delay() for i in range(4)]
然后我希望拥有--concurrency=1
参数的动态工作人数。这对我的情况至关重要,因为每个工作者只有一个并行任务,并且当某些任务在服务器上排队时添加工作者。如果某些任务没有其工作人员并且添加了新工作程序,则新工作人员应该处理排队(待处理但未执行)任务。但是当我打电话给$ celery -A case worker --loglevel=info --concurrency=1
时,我有以下日志(不包括Celery徽标):
[tasks]
. case.do_sth
[2017-02-21 10:03:18,454: INFO/MainProcess] Connected to redis://localhost:6379//
[2017-02-21 10:03:18,459: INFO/MainProcess] mingle: searching for neighbors
[2017-02-21 10:03:19,471: INFO/MainProcess] mingle: all alone
[2017-02-21 10:03:19,480: INFO/MainProcess] celery@pt0 ready.
[2017-02-21 10:03:19,658: INFO/MainProcess] Received task: case.do_sth[0c7b0f8c-d1f8-4cd8-a100-21ef6654e04c]
[2017-02-21 10:03:19,660: INFO/MainProcess] Received task: case.do_sth[f97ad614-017b-4a6c-90df-89dbed63e39b]
[2017-02-21 10:03:19,662: INFO/MainProcess] Received task: case.do_sth[b0166022-196f-451b-bcb6-78cdf0558803]
[2017-02-21 10:03:19,664: INFO/MainProcess] Received task: case.do_sth[b097e191-5bc4-44d9-bdcd-8aa74501e95d]
[2017-02-21 10:03:24,667: INFO/PoolWorker-1] Task case.do_sth[0c7b0f8c-d1f8-4cd8-a100-21ef6654e04c] succeeded in 5.006301835s: None
[2017-02-21 10:03:29,675: INFO/PoolWorker-1] Task case.do_sth[f97ad614-017b-4a6c-90df-89dbed63e39b] succeeded in 5.005384011s: None
[2017-02-21 10:03:34,683: INFO/PoolWorker-1] Task case.do_sth[b0166022-196f-451b-bcb6-78cdf0558803] succeeded in 5.005373027s: None
[2017-02-21 10:03:39,690: INFO/PoolWorker-1] Task case.do_sth[b097e191-5bc4-44d9-bdcd-8aa74501e95d] succeeded in 5.00531687s: None
与此同时,我开始了另一个工人(使用相同的命令),但它没有做任何事情:
[tasks]
. case.do_sth
[2017-02-21 10:03:20,321: INFO/MainProcess] Connected to redis://localhost:6379//
[2017-02-21 10:03:20,326: INFO/MainProcess] mingle: searching for neighbors
[2017-02-21 10:03:21,339: INFO/MainProcess] mingle: all alone
[2017-02-21 10:03:21,352: INFO/MainProcess] celery@pt0 ready.
如果你查看日期,你可以看到第二个工人在第一个工人被触发后不到五秒钟。
是否有办法(某些工人或芹菜选项)将单个工作人员排队的任务限制为一个任务?