Question

我想安排一项任务在将来的不同时间运行。我在调用任务时尝试使用eta选项来执行此操作。这部分可以自行运行，但与此同时，我希望对任务有一个速率限制。我无法做到这两点：安排任务在将来运行，但也要让芹菜尊重速率限制。

这是一个例子（rate_limit_eta.py）

from celery import Celery
from datetime import datetime, timedelta

app = Celery('rate_limit_eta', broker='amqp://guest@localhost//')

@app.task(rate_limit='4/m')
def my_task():
    print 'execution timestamp: {}'.format(datetime.utcnow())


if __name__ == '__main__':

    now = datetime.utcnow()
    d = now + timedelta(seconds=5)
    print now

    for i in range(10):
        my_task.apply_async(eta=d)

在循环中，当我打电话给'my_task＆＃39;使用eta = d，任务尽可能快地执行。不尊重rate_limit。

my_task.apply_async(eta=d)


[2015-06-12 13:45:15,750: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.748498
[2015-06-12 13:45:15,757: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.757757
[2015-06-12 13:45:15,760: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.759992
[2015-06-12 13:45:15,763: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.763614
[2015-06-12 13:45:15,766: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.765996
[2015-06-12 13:45:15,768: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.768424
[2015-06-12 13:45:15,771: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.771079
[2015-06-12 13:45:15,774: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.774561
[2015-06-12 13:45:15,777: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.777425
[2015-06-12 13:45:15,780: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.780207

但如果我没有安排任务（即没有eta），则强制执行速率限制。

my_task.apply_async()

[2015-06-12 13:47:05,844: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.843872
[2015-06-12 13:47:05,850: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.850502
[2015-06-12 13:47:21,919: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:21.919120
[2015-06-12 13:47:35,937: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:35.937060
[2015-06-12 13:47:51,959: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:51.959174
[2015-06-12 13:48:05,976: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:05.976197
[2015-06-12 13:48:21,998: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:21.998254
[2015-06-12 13:48:36,014: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:36.014837
[2015-06-12 13:48:52,041: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:52.041326
[2015-06-12 13:49:06,057: WARNING/Worker-1] execution timestamp: 2015-06-12 13:49:06.057286

我做错了吗？

Answer 1

我有同样的问题，我找到了原因。我查看了Celery代码，这种行为来自the strategy代码，它执行以下操作：

if req.eta:
    apply_eta()
else:
    check_if_rate_limited()

我担心这样做ETA和速率限制是两种不兼容的方式告诉Celery什么时候应该执行任务。芹菜必须选择一种，而恰好是ETA。

我的链接是Celery的v3（因为我怀疑这个问题被要求提供v3），但在撰写本文时，这仍然适用于Celery的主分支。我想这个限制可以在某处更好地记录（here?），它只是一个拉取请求！

更新：我在Celery项目上打开了an issue。

通过＆＃39; eta＆＃39;进行日程安排时，Celery rate_limit不受尊重。选项

1 个答案: