我们可以通过RabbitMQ加速发布消息

时间:2016-03-09 14:36:29

标签: python-3.x rabbitmq ubuntu-14.04

我正在我的Ubuntu工作站上运行一些测试。这些基准测试从填充队列开始,该队列运行速度非常慢:

import pika
import datetime

if __name__ == '__main__':
    try:
        connection = pika.BlockingConnection(pika.ConnectionParameters(
                host='localhost'))
        channel = connection.channel()

        channel.queue_declare(queue='hello_durable', durable=True)
        started_at = datetime.datetime.now()
        properties = pika.BasicProperties(delivery_mode=2)
        for i in range(0, 100000):
            channel.basic_publish(exchange='',
                                  routing_key='hello',
                                  body='Hello World!',
                                  properties=properties)
            if i%10000 == 0:
                duration = datetime.datetime.now() - started_at
                print(i, duration.total_seconds())
        print(" [x] Sent 'Hello World!'")
        connection.close()
        now = datetime.datetime.now()
        duration = now - started_at
        print(duration.total_seconds())
    except Exception as e:
        print(e)

发送10K消息需要30多秒。根据top命令,工作站有12个核心,它们并不繁忙。有超过8Gb的可用内存。队列是否持久并不重要。

我们如何加快发送邮件的速度?

2 个答案:

答案 0 :(得分:2)

从BlockingConnection切换到SelectConnection产生了巨大的差异,加快了这个过程近五十倍。我需要做的就是修改the following tutorial:中的示例,在循环中发布消息:

import pika

# Step #3
def on_open(connection):

    connection.channel(on_channel_open)

# Step #4
def on_channel_open(channel):

    channel.basic_publish('test_exchange',
                            'test_routing_key',
                            'message body value',
                            pika.BasicProperties(content_type='text/plain',
                                                 delivery_mode=1))

    connection.close()

# Step #1: Connect to RabbitMQ
parameters = pika.URLParameters('amqp://guest:guest@localhost:5672/%2F')

connection = pika.SelectConnection(parameters=parameters,
                                   on_open_callback=on_open)

try:

    # Step #2 - Block on the IOLoop
    connection.ioloop.start()

# Catch a Keyboard Interrupt to make sure that the connection is closed cleanly
except KeyboardInterrupt:

    # Gracefully close the connection
    connection.close()

    # Start the IOLoop again so Pika can communicate, it will stop on its own when the connection is closed
    connection.ioloop.start()

答案 1 :(得分:1)

假设您没有运行任何消费者These benchmarks start with populating a queue。 由于您只发布消息,因此rabbitmq切换到流状态。更准确地说,您的交换和/或队列将进入流状态。 引自rabbitmq blog

  

这(大致)意味着客户受到速率限制;它会   喜欢发布得更快,但服务器无法跟上

我敢肯定,如果你看得足够近,你会看到消息的第一部分(在初始设置时,空队列)变快,但发送速率在某些时候急剧下降。