如何在celery task.apply_async中使用优先级

时间:2017-12-26 14:17:18

标签: python rabbitmq queue celery priority-queue

我在芹菜中有一个<div class="download-section"> ... <div class="panel-body" style="max-height: 300px;overflow-y: scroll;"> <ul class="list-group" id="ulid"> <?php include 'myextraction.php';?> </ul> </div> <script src="https://code.jquery.com/jquery-1.10.2.js"> $('#ulid').scrollTop($('#ulid li:nth-child(#)').position().top); </script> ... </div> 队列,我为它定义了一个任务:

test

只打印参数。我想设置为@celery_app.task(queue='test', ignore_result=True) def priority_test(priority): print(priority) 定义herepriority属性。所以,我写了appy_async这样的话:

for loop

我想看到这样的结果:

for i in range(100):
    priority_test.apply_async((i%10,), queue="test", priority=i%10)

意味着在彼此之后执行相同的优先级,但它以正常方式执行它们:

[2017-12-26 17:21:37,309: WARNING/ForkPoolWorker-1] 10
[2017-12-26 17:21:37,311: WARNING/ForkPoolWorker-1] 10
[2017-12-26 17:21:37,314: WARNING/ForkPoolWorker-1] 10
[2017-12-26 17:21:37,317: WARNING/ForkPoolWorker-1] 9
[2017-12-26 17:21:37,319: WARNING/ForkPoolWorker-1] 9
[2017-12-26 17:21:37,321: WARNING/ForkPoolWorker-1] 9
[2017-12-26 17:21:37,323: WARNING/ForkPoolWorker-1] 8
[2017-12-26 17:21:37,326: WARNING/ForkPoolWorker-1] 8
[2017-12-26 17:21:37,329: WARNING/ForkPoolWorker-1] 8
[2017-12-26 17:21:37,332: WARNING/ForkPoolWorker-1] 7
[2017-12-26 17:21:37,334: WARNING/ForkPoolWorker-1] 7
[2017-12-26 17:21:37,336: WARNING/ForkPoolWorker-1] 7
[2017-12-26 17:21:37,341: WARNING/ForkPoolWorker-1] 6
[2017-12-26 17:21:37,344: WARNING/ForkPoolWorker-1] 6
[2017-12-26 17:21:37,346: WARNING/ForkPoolWorker-1] 6
[2017-12-26 17:21:37,349: WARNING/ForkPoolWorker-1] 5
[2017-12-26 17:21:37,351: WARNING/ForkPoolWorker-1] 5
[2017-12-26 17:21:37,353: WARNING/ForkPoolWorker-1] 5
[2017-12-26 17:21:37,355: WARNING/ForkPoolWorker-1] 4
[2017-12-26 17:21:37,358: WARNING/ForkPoolWorker-1] 4
[2017-12-26 17:21:37,360: WARNING/ForkPoolWorker-1] 4

我应该如何在芹菜中使用rabbitmq申请[2017-12-26 17:21:37,309: WARNING/ForkPoolWorker-1] 10 [2017-12-26 17:21:37,311: WARNING/ForkPoolWorker-1] 9 [2017-12-26 17:21:37,314: WARNING/ForkPoolWorker-1] 8 [2017-12-26 17:21:37,317: WARNING/ForkPoolWorker-1] 7 [2017-12-26 17:21:37,319: WARNING/ForkPoolWorker-1] 6 [2017-12-26 17:21:37,321: WARNING/ForkPoolWorker-1] 5 [2017-12-26 17:21:37,323: WARNING/ForkPoolWorker-1] 4 [2017-12-26 17:21:37,326: WARNING/ForkPoolWorker-1] 3 [2017-12-26 17:21:37,329: WARNING/ForkPoolWorker-1] 2 [2017-12-26 17:21:37,332: WARNING/ForkPoolWorker-1] 1 [2017-12-26 17:21:37,334: WARNING/ForkPoolWorker-1] 10 [2017-12-26 17:21:37,336: WARNING/ForkPoolWorker-1] 9 [2017-12-26 17:21:37,341: WARNING/ForkPoolWorker-1] 8 [2017-12-26 17:21:37,344: WARNING/ForkPoolWorker-1] 7 [2017-12-26 17:21:37,346: WARNING/ForkPoolWorker-1] 6 [2017-12-26 17:21:37,349: WARNING/ForkPoolWorker-1] 5 [2017-12-26 17:21:37,351: WARNING/ForkPoolWorker-1] 4 [2017-12-26 17:21:37,353: WARNING/ForkPoolWorker-1] 3 [2017-12-26 17:21:37,355: WARNING/ForkPoolWorker-1] 2 [2017-12-26 17:21:37,358: WARNING/ForkPoolWorker-1] 1 [2017-12-26 17:21:37,360: WARNING/ForkPoolWorker-1] 10 [2017-12-26 17:21:37,362: WARNING/ForkPoolWorker-1] 9 [2017-12-26 17:21:37,364: WARNING/ForkPoolWorker-1] 8 [2017-12-26 17:21:37,365: WARNING/ForkPoolWorker-1] 7 [2017-12-26 17:21:37,367: WARNING/ForkPoolWorker-1] 6 [2017-12-26 17:21:37,369: WARNING/ForkPoolWorker-1] 5 [2017-12-26 17:21:37,371: WARNING/ForkPoolWorker-1] 4 [2017-12-26 17:21:37,373: WARNING/ForkPoolWorker-1] 3 [2017-12-26 17:21:37,374: WARNING/ForkPoolWorker-1] 2 [2017-12-26 17:21:37,376: WARNING/ForkPoolWorker-1] 1 以及上述文档中的priority属性是什么?

1 个答案:

答案 0 :(得分:3)

为了让priority正常工作,您需要正确配置几个设置,并且至少需要3.5.0版本的RabbitMQ。

首先将队列的x-max-priority设置为10.来自docs

from kombu import Exchange, Queue

app.conf.task_queues = [
    Queue('tasks', Exchange('tasks'), routing_key='tasks',
          queue_arguments={'x-max-priority': 10},
]

可以使用task_queue_max_priority设置设置所有队列的默认值:

app.conf.task_queue_max_priority = 10

然后配置以下设置:

CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

默认情况下,预取乘数为4,在您的情况下,将导致在队列中存在其他任务之前获取优先级为10,9,8和7的前4个任务。 CELERY_ACKS_LATE设置将导致任务在执行后得到确认。您可以尝试使用此设置来查看您喜欢的行为。