RabbitMQ等待多个队列完成

Question

好的，这是对正在发生的事情的概述：

    M <-- Message with unique id of 1234
    |
    +-Start Queue
    |
    |
    | <-- Exchange
   /|\
  / | \
 /  |  \ <-- bind to multiple queues
Q1  Q2  Q3
\   |   / <-- start of the problem is here
 \  |  / 
  \ | /
   \|/
    |
    Q4 <-- Queues 1,2 and 3 must finish first before Queue 4 can start
    |
    C <-- Consumer 

所以我有一个推送到多个队列的交换，每个队列都有一个任务，一旦所有任务完成，只有队列4才能启动。

所以具有唯一ID 1234的消息被发送到交换机，交换机将其路由到所有任务队列（Q1，Q2，Q3等...），当消息ID 1234的所有任务都已完成时，运行消息ID 1234的Q4。

我该如何实现？

使用Symfony2，RabbitMQBundle和RabbitMQ 3.x

资源：

• http://www.rabbitmq.com/tutorials/amqp-concepts.html
• http://www.rabbitmq.com/tutorials/tutorial-six-python.html

更新＃1

好的我觉得这就是我要找的东西：

• https://github.com/videlalvaro/Thumper/tree/master/examples/parallel_processing

RPC with Parallel Processing，但是如何将Correlation Id设置为我的唯一ID来对邮件进行分组并确定哪个队列？

Answer 1

你需要实现这个：http://www.eaipatterns.com/Aggregator.html但是针对Symfony的RabbitMQBundle不支持，所以你必须使用底层的php-amqplib。

来自捆绑包的普通消费者回调将获得AMQPMessage。从那里，您可以访问该频道并手动发布到“管道和过滤器”实现中的下一个交换

Answer 2

在RabbitMQ网站的RPC教程中，有一种方法可以传递“关联ID”，可以识别队列中用户的消息。

我建议在前三个队列中使用某种带有消息的id，然后再使用另一个进程将消息从3中排队到某种类型的桶中。当那些桶收到我假设完成的3个任务时，将最终消息发送到第4个队列进行处理。

如果要为一个用户向每个队列发送多个工作项，则可能需要进行一些预处理以找出特定用户放入队列的项目数，以便在4之前出列的进程知道有多少项在排队之前期待。

---

我在C＃中做了我的rabbitmq，很抱歉我的伪代码不是php风格

// Client
byte[] body = new byte[size];
body[0] = uniqueUserId;
body[1] = howManyWorkItems;
body[2] = command;

// Setup your body here

Queue(body)

---

// Server
// Process queue 1, 2, 3
Dequeue(message)

switch(message.body[2])
{
    // process however you see fit
}

processedMessages[message.body[0]]++;

if(processedMessages[message.body[0]] == message.body[1])
{
    // Send to queue 4
    Queue(newMessage)
}

---

对更新＃1的响应

不要将客户端视为终端，将客户端视为服务器上的进程可能会有所帮助。因此，如果您在this one之类的服务器上设置RPC客户端，那么您需要做的就是让服务器处理用户唯一ID的生成并将消息发送到适当的队列：

    public function call($uniqueUserId, $workItem) {
        $this->response = null;
        $this->corr_id = uniqid();

        $msg = new AMQPMessage(
            serialize(array($uniqueUserId, $workItem)),
            array('correlation_id' => $this->corr_id,
            'reply_to' => $this->callback_queue)
        );

        $this->channel->basic_publish($msg, '', 'rpc_queue');
        while(!$this->response) {
            $this->channel->wait();
        }

        // We assume that in the response we will get our id back
        return deserialize($this->response);
    }


$rpc = new Rpc();

// Get unique user information and work items here

// Pass even more information in here, like what queue to use or you could even loop over this to send all the work items to the queues they need.
$response = rpc->call($uniqueUserId, $workItem);

$responseBuckets[array[0]]++;

// Just like above code that sees if a bucket is full or not

Answer 3

我有点不清楚你想要在这里实现什么。但是我可能会稍微改变一下设计，以便一旦从队列中清除所有消息，就会发布到一个单独的交换机，该交换机将发布到队列4中。

Answer 4

除了RPC based answer之外，我还要添加另一个基于EIP aggregator pattern的内容。

接下来的想法是：一切都是异步的，没有RPC或其他同步的东西。每个任务在完成时发送，聚合器订阅该事件。它基本上计算任务并在计数器达到预期数量时发送task4消息（在我们的例子中为3）。我选择一个文件系统作为计数器的存储空间来简化。你可以在那里使用数据库。

制作人看起来更简单。它只会激发并忘记

<?php
use Enqueue\Client\Message;
use Enqueue\Client\ProducerInterface;
use Enqueue\Util\UUID;
use Symfony\Component\DependencyInjection\ContainerInterface;

/** @var ContainerInterface $container */

/** @var ProducerInterface $producer */
$producer = $container->get('enqueue.client.producer');

$message = new Message('the task data');
$message->setCorrelationId(UUID::generate());

$producer->sendCommand('task1', clone $message);
$producer->sendCommand('task2', clone $message);
$producer->sendCommand('task3', clone $message);

任务处理器必须在其工作完成后发送事件：

<?php
use Enqueue\Client\CommandSubscriberInterface;
use Enqueue\Client\Message;
use Enqueue\Client\ProducerInterface;
use Enqueue\Psr\PsrContext;
use Enqueue\Psr\PsrMessage;
use Enqueue\Psr\PsrProcessor;

class Task1Processor implements PsrProcessor, CommandSubscriberInterface
{
    private $producer;

    public function __construct(ProducerInterface $producer)
    {
        $this->producer = $producer;
    }

    public function process(PsrMessage $message, PsrContext $context)
    {
        // do the job

        // same for other
        $eventMessage = new Message('the event data');
        $eventMessage->setCorrelationId($message->getCorrelationId());

        $this->producer->sendEvent('task_is_done', $eventMessage);

        return self::ACK;
    }

    public static function getSubscribedCommand()
    {
        return 'task1';
    }
}

聚合器处理器：

<?php

use Enqueue\Client\TopicSubscriberInterface;
use Enqueue\Psr\PsrContext;
use Enqueue\Psr\PsrMessage;
use Enqueue\Psr\PsrProcessor;
use Symfony\Component\Filesystem\LockHandler;

class AggregatorProcessor implements PsrProcessor, TopicSubscriberInterface
{
    private $producer;
    private $rootDir;

    /**
     * @param ProducerInterface $producer
     * @param string $rootDir
     */
    public function __construct(ProducerInterface $producer, $rootDir)
    {
        $this->producer = $producer;
        $this->rootDir = $rootDir;
    }

    public function process(PsrMessage $message, PsrContext $context)
    {
        $expectedNumberOfTasks = 3;

        if (false == $cId = $message->getCorrelationId()) {
            return self::REJECT;
        }

        try {
            $lockHandler = new LockHandler($cId, $this->rootDir.'/var/tasks');
            $lockHandler->lock(true);

            $currentNumberOfProcessedTasks = 0;
            if (file_exists($this->rootDir.'/var/tasks/'.$cId)) {
                $currentNumberOfProcessedTasks = file_get_contents($this->rootDir.'/var/tasks/'.$cId);

                if ($currentNumberOfProcessedTasks +1 == $expectedNumberOfTasks) {
                    unlink($this->rootDir.'/var/tasks/'.$cId);

                    $this->producer->sendCommand('task4', 'the task data');

                    return self::ACK;
                }
            }

            file_put_contents($this->rootDir.'/var/tasks/'.$cId, ++$currentNumberOfProcessedTasks);

            return self::ACK;
        } finally {
            $lockHandler->release();
        }
    }

    public static function getSubscribedTopics()
    {
        return 'task_is_done';
    }
}

Answer 5

我可以告诉你如何使用enqueue-bundle来完成它。

所以用composer安装它并注册为任何其他bundle。然后配置：

    var p=document.getElementById('childId').parentNode;
    var c=document.getElementById('childId');
    p.removeChild(c);
    alert('Deleted');

此方法基于RPC。这是你如何做到的：

// app/config/config.yml

enqueue:
  transport:
    default: 'amnqp://'
  client: ~

消费者处理器如下所示：

<?php
use Enqueue\Client\ProducerInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

/** @var ContainerInterface $container */

/** @var ProducerInterface $producer */
$producer = $container->get('enqueue.client.producer');

$promises = new SplObjectStorage();

$promises->attach($producer->sendCommand('task1', 'the task data', true));
$promises->attach($producer->sendCommand('task2', 'the task data', true));
$promises->attach($producer->sendCommand('task3', 'the task data', true));

while (count($promises)) {
    foreach ($promises as $promise) {
        if ($replyMessage = $promise->receiveNoWait()) {
            // you may want to check the response here
            $promises->detach($promise);
        }
    }
}

$producer->sendCommand('task4', 'the task data');

将其作为带有标记use Enqueue\Client\CommandSubscriberInterface; use Enqueue\Consumption\Result; use Enqueue\Psr\PsrContext; use Enqueue\Psr\PsrMessage; use Enqueue\Psr\PsrProcessor; class Task1Processor implements PsrProcessor, CommandSubscriberInterface { public function process(PsrMessage $message, PsrContext $context) { // do task job return Result::reply($context->createMessage('the reply data')); } public static function getSubscribedCommand() { // you can simply return 'task1'; if you do not need a custom queue, and you are fine to use what enqueue chooses. return [ 'processorName' => 'task1', 'queueName' => 'Q1', 'queueNameHardcoded' => true, 'exclusive' => true, ]; } } 的服务添加到您的容器中，然后运行命令enqueue.client.processor

Here's the plain PHP version