Question

我有一个RabbitMQ（版本3.2.4）异步使用者（如所描述的here）实现并侦听队列/路由键，并且在我最近做了一些更改之前一直运行没有任何问题。

某些任务非常耗时，因此我决定使用多处理库来分离使用多处理队列/池设计执行这些密集任务的子流程，以便我的主要任务无需等待即可执行。

my_queue = multiprocessing.Queue()
my_pool = multiprocessing.Pool(2, my_method, (my_queue,))

初始化队列和池后，我将队列作为参数传递，同时初始化使用者（ExampleConsumer＆＃39; __init__方法，如上面的示例链接所示。然后，在on_message方法中，我将消息推送到my_queue以执行时间密集型任务。

修改

一些代码示例：

def main():
    logging.basicConfig(level=logging.INFO, format=LOG_FORMAT)
    my_queue = multiprocessing.Queue()
    my_pool = multiprocessing.Pool(2, my_class().my_method, (my_queue,))
    example = ExampleConsumer('amqp://guest:guest@localhost:5672/%2F', my_queue)
    try:
        example.run()
        my_pool.close()
        my_pool.join()
    except KeyboardInterrupt:
        my_pool.terminate()
        example.stop()

消费者的init方法和on_message方法：

def __init__(self, amqp_url, queue):
        """Create a new instance of the consumer class, passing in the AMQP
        URL used to connect to RabbitMQ.

        :param str amqp_url: The AMQP url to connect with

        """
        self._connection = None
        self._channel = None
        self._closing = False
        self._consumer_tag = None
        self._url = amqp_url
        self.queue = queue

def on_message(self, unused_channel, basic_deliver, properties, body):
        """Invoked by pika when a message is delivered from RabbitMQ. The
        channel is passed for your convenience. The basic_deliver object that
        is passed in carries the exchange, routing key, delivery tag and
        a redelivered flag for the message. The properties passed in is an
        instance of BasicProperties with the message properties and the body
        is the message that was sent.

        :param pika.channel.Channel unused_channel: The channel object
        :param pika.Spec.Basic.Deliver: basic_deliver method
        :param pika.Spec.BasicProperties: properties
        :param str|unicode body: The message body

        """
        LOGGER.info('Received message # %s from %s: %s',
                    basic_deliver.delivery_tag, properties.app_id, body)
        self.acknowledge_message(basic_deliver.delivery_tag)
        self.queue.put(str(body))

进行这些更改后，我开始看到以下类型的异常：

File "consumer_new.py", line 500, in run
    self._connection.ioloop.start()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 355, in start
    self.process_timeouts()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 283, in process_timeouts
    timer['callback']()
  File "consumer_new.py", line 290, in reconnect
    self._connection.ioloop.start()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 354, in start
    self.poll()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 602, in poll
    self._process_fd_events(fd_event_map, write_only)
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events
    handler(fileno, events, write_only=write_only)
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 364, in _handle_events
    self._handle_read()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 415, in _handle_read
    self._on_data_available(data)
  File "/usr/local/lib/python2.7/site-packages/pika/connection.py", line 1347, in _on_data_available
    self._process_frame(frame_value)
  File "/usr/local/lib/python2.7/site-packages/pika/connection.py", line 1427, in _process_frame
    self._deliver_frame_to_channel(frame_value)
  File "/usr/local/lib/python2.7/site-packages/pika/connection.py", line 1028, in _deliver_frame_to_channel
    return self._channels[value.channel_number]._handle_content_frame(value)
  File "/usr/local/lib/python2.7/site-packages/pika/channel.py", line 896, in _handle_content_frame
    self._on_deliver(*response)
  File "/usr/local/lib/python2.7/site-packages/pika/channel.py", line 983, in _on_deliver
    header_frame.properties, body)
  File "consumer_new.py", line 452, in on_message
    self.acknowledge_message(basic_deliver.delivery_tag)
  File "consumer_new.py", line 463, in acknowledge_message
    self._channel.basic_ack(delivery_tag)
  File "/usr/local/lib/python2.7/site-packages/pika/channel.py", line 159, in basic_ack
    return self._send_method(spec.Basic.Ack(delivery_tag, multiple))
  File "/usr/local/lib/python2.7/site-packages/pika/channel.py", line 1150, in _send_method
    self.connection._send_method(self.channel_number, method_frame, content)
  File "/usr/local/lib/python2.7/site-packages/pika/connection.py", line 1569, in _send_method
    self._send_frame(frame.Method(channel_number, method_frame))
  File "/usr/local/lib/python2.7/site-packages/pika/connection.py", line 1554, in _send_frame
    self._flush_outbound()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 282, in _flush_outbound
    self._handle_write()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 452, in _handle_write
    return self._handle_error(error)
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 338, in _handle_error
    self._handle_disconnect()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 288, in _handle_disconnect
    self._adapter_disconnect()
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 94, in _adapter_disconnect
    self.ioloop.remove_handler(self.socket.fileno())
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 579, in remove_handler
    super(PollPoller, self).remove_handler(fileno)
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 328, in remove_handler
    self.update_handler(fileno, 0)
  File "/usr/local/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 571, in update_handler
    self._poll.modify(fileno, events)
IOError: [Errno 9] Bad file descriptor

run()方法在主进程中继续运行，无需任何干预。如果是这种情况我不明白为什么会出现错误的文件描述符错误，因为没有其他人可以关闭rmq连接。此外，由于上述原因，消费者在失败前3-4小时似乎没有任何问题。

如果文件描述符数量不足，我检查了Rabbitmq UI。但这似乎不是问题所在。我无法在可能出现的问题上取得领先。

任何帮助表示赞赏！感谢。

Answer 1

Pika不是线程安全的。它在文档中清楚地说明了。如果您对线程或子进程中的连接或通道执行任何操作，那么各种各样的事情最终都会出错并且您的程序将崩溃到奇怪且无法提供信息的错误。它可能似乎工作了一段时间，但最终Pika结构将被破坏。

如果您需要多处理和rabbitmq，您有几个选择。

使用rabbitpy而不是Pika。我没有使用它所以我不能评论它对你的适用性，但它是线程安全的。
如果可以，请分开任务，以便您的子流程可以打开自己的Pika连接。如果您的主程序收到请求，有一个子进程来处理它然后发送结果，这不起作用。例如，如果您需要发送确认，则不能在主进程中收到子进程确认消息。
从子流程中删除Pika。如果您的子进程的想法是将计算或耗时的任务分派给它们，您可以尝试创建两个队列：一个用于子进程输入，一个用于输出，并让您的子进程将结果返回到队列中的主程序。然后主程序可以基于此处理rabbitmq流量。
如果您的程序是处理请求的某种服务器，则将所有内容拆分为子进程（“工作队列” - 模型）https://www.rabbitmq.com/tutorials/tutorial-two-python.html，并让每个子进程独立订阅作为队列的使用者。 Rabbitmq负责循环调度，并且通过限制预取，你可以使它成为一个子进程只选择一个任务，直到完成该任务的处理，它将不会拾取任何其他东西，确保在第一个任务之后立即发送任务一个将由空闲线程或子进程拾取。在这个模型中，你的主要根本不需要Pika连接，每个子进程都有一个独立的连接，如2）所示。

希望这会有所帮助。

哈努哈利

RMQ异步使用者：pika.adapters.base_connection，_handle_error，致命套接字错误：错误（9，＆＃39;错误的文件描述符＆＃39;）

1 个答案: