使用spring-amqp和rabbitmq实现带有退避的非阻塞重试

时间：2015-09-16 20:56:43

标签： rabbitmq spring-amqp spring-retry

我正在寻找使用spring amqp和Rabbit MQ实现带有退避策略的重试的好方法，但要求是不应该阻止侦听器（因此可以自由处理其他消息）。我在这里看到了一个类似的问题/答案，但它没有包括'退出'的解决方案：

RabbitMQ & Spring amqp retry without blocking consumers

我的问题是：

  1. 重试时，默认的spring-retry实现会阻塞线程吗？ implementation in github表示确实如此。

  2. 如果上面的假设是真的，那么实现这个实现单独的重试队列（DLQ？），并为每条消息设置一个TTL的唯一方法（假设我们不想阻止线程的退避间隔）。

  3. 如果我们采用上述方法（DLQ或单独的队列），我们是否需要为每次重试尝试使用单独的队列？如果我们只使用1个队列进行重试，则同一队列将包含TTL范围从最小重试间隔到最大重试间隔的消息，如果队列前面的消息具有最大TTL，则其后面的消息将不会是即使它有最小TTL也会被拿起。这是根据Rabbit MQ TTL文档here（参见警告）：

  4. 有没有其他方法可以实现非阻塞退避重试机制？

    5. 添加一些配置信息以帮助解决@garyrussel问题：

    队列配置：

        <rabbit:queue name="regular_requests_queue"/>
    <rabbit:queue name="retry_requests_queue">
        <rabbit:queue-arguments>
            <entry key="x-dead-letter-exchange" value="regular_exchange" />
        </rabbit:queue-arguments>
    </rabbit:queue>

    <rabbit:direct-exchange name="regular_exchange">
        <rabbit:bindings>
            <rabbit:binding queue="regular_requests_queue" key="regular-request-key"/>
        </rabbit:bindings>
    </rabbit:direct-exchange>

    <rabbit:direct-exchange name="retry_exchange">
        <rabbit:bindings>
            <rabbit:binding queue="retry_requests_queue"/>
        </rabbit:bindings>
    </rabbit:direct-exchange>

    <bean id="retryRecoverer" class="com.testretry.RetryRecoverer">
         <constructor-arg ref="retryTemplate"/>
         <constructor-arg value="retry_exchange"/>
    </bean>

    <rabbit:template id="templateWithOneRetry" connection-factory="connectionFactory" exchange="regular_exchange" retry-template="retryTemplate"/>
    <rabbit:template id="retryTemplate" connection-factory="connectionFactory" exchange="retry_exchange"/>

    <bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
        <property name="retryPolicy">
            <bean class="org.springframework.retry.policy.SimpleRetryPolicy">
                <property name="maxAttempts" value="1"/>
            </bean>
        </property>
    </bean>

3 个答案:

答案 0 :(得分：0)

  1. 是
  2. 到4 ......

    3. 您可以使用子类RepublishMessageRecoverer重试max attempts = 1并实现additionalHeaders来添加，例如重试计数标题。

    然后，您可以为每次尝试重新发布到不同的队列。

    恢复器的结构并不是为了发布到不同的队列（我们应该更改它），因此您可能需要编写自己的恢复器并委托给多个RepublishMessageRecoverer中的一个。

    考虑contributing您对框架的解决方案。

答案 1 :(得分：0)

这是我最终实施的最终解决方案。每个“重试间隔”有1个队列，每个重试队列有1个交换。它们都被传递给自定义的RepublishRecoverer，它创建了一个恢复者列表。

将一个名为“RetryCount”的自定义标头添加到消息中，并根据“RetryCount”的值，将消息发布到具有不同“过期”的正确交换/队列。每个重试队列都使用DLX设置，DLX设置为'regular_exchange'（即请求转到常规队列）。

<rabbit:template id="genericTemplateWithRetry" connection-factory="connectionFactory" exchange="regular_exchange" retry-template="retryTemplate"/>

<!-- Create as many templates as retryAttempts (1st arg) in customRetryTemplate-->
<rabbit:template id="genericRetryTemplate1" connection-factory="consumerConnFactory" exchange="retry_exchange_1"/>
<rabbit:template id="genericRetryTemplate2" connection-factory="consumerConnFactory" exchange="retry_exchange_2"/>
<rabbit:template id="genericRetryTemplate3" connection-factory="consumerConnFactory" exchange="retry_exchange_3"/>
<rabbit:template id="genericRetryTemplate4" connection-factory="consumerConnFactory" exchange="retry_exchange_4"/>
<rabbit:template id="genericRetryTemplate5" connection-factory="consumerConnFactory" exchange="retry_exchange_5"/>

<rabbit:queue name="regular_requests_queue"/>

<!-- Create as many queues as retryAttempts (1st arg) in customRetryTemplate -->
<rabbit:queue name="retry_requests_queue_1">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_2">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_3">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_4">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_5">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>

<rabbit:direct-exchange name="regular_exchange">
    <rabbit:bindings>
        <rabbit:binding queue="regular_requests_queue" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<!-- Create as many exchanges as retryAttempts (1st arg) in customRetryTemplate -->
<rabbit:direct-exchange name="retry_exchange_1">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_1" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_2">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_2" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_3">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_3" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_4">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_4" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_5">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_5" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>


<!-- retry config begin -->
<!-- Pass in all templates and exchanges created as list/array arguments below -->
<bean id="customRetryRecoverer" class="com.test.listeners.CustomRetryRecoverer">
    <!-- Pass in list of templates -->
     <constructor-arg>
        <list>
            <ref bean="genericRetryTemplate1"/>
            <ref bean="genericRetryTemplate2"/>
            <ref bean="genericRetryTemplate3"/>
            <ref bean="genericRetryTemplate4"/>
            <ref bean="genericRetryTemplate5"/>
        </list>
     </constructor-arg>
     <!-- Pass in array of exchanges -->
     <constructor-arg value="retry_exchange_1,retry_exchange_2,retry_exchange_3,retry_exchange_4,retry_exchange_5"/>
     <constructor-arg ref="customRetryTemplate"/>
</bean>

<bean id="retryInterceptor"
      class="org.springframework.amqp.rabbit.config.StatefulRetryOperationsInterceptorFactoryBean">
    <property name="messageRecoverer" ref="customRetryRecoverer"/>
    <property name="retryOperations" ref="retryTemplate"/>
    <property name="messageKeyGenerator" ref="msgKeyGenerator"/>
</bean>

<bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
    <property name="retryPolicy">
        <bean class="org.springframework.retry.policy.SimpleRetryPolicy">
            <!--  Set to 1 - just for the initial attempt -->
            <property name="maxAttempts" value="1"/>
        </bean>
    </property>
</bean>

 <bean id="customRetryTemplate" class="com.test.retry.CustomRetryTemplate">
    <constructor-arg value="5"/> <!-- max attempts -->
    <constructor-arg value="3000"/> <!-- Initial interval -->
    <constructor-arg value="5"/> <!-- multiplier for backoff -->
</bean>

<!-- retry config end -->

以下是CustomRetryRecoverer的代码：

public class CustomRetryRecoverer extends
        RepublishMessageRecoverer {

    private static final String RETRY_COUNT_HEADER_NAME = "RetryCount";
    private List<RepublishMessageRecoverer> retryExecutors = new ArrayList<RepublishMessageRecoverer>();
    private TriggersRetryTemplate retryTemplate;

    public TriggersRetryRecoverer(AmqpTemplate[] retryTemplates, String[] exchangeNames, TriggersRetryTemplate retryTemplate) {
        super(retryTemplates[0], exchangeNames[0]);
        this.retryTemplate = retryTemplate;

        //Get lower of the two array sizes
        int executorCount = (exchangeNames.length < retryTemplates.length) ? exchangeNames.length : retryTemplates.length;
        for(int i=0; i<executorCount; i++) {
            createRetryExecutor(retryTemplates[i], exchangeNames[i]);
        }
        //If not enough exchanges/templates provided, reuse the last exchange/template for the remaining retry recoverers
        if(retryTemplate.getMaxRetryCount() > executorCount) {
            for(int i=executorCount; i<retryTemplate.getMaxRetryCount(); i++) {
                createRetryExecutor(retryTemplates[executorCount-1], exchangeNames[executorCount-1]);
            }
        }
    }

    @Override
    public void recover(Message message, Throwable cause) {

        if(getRetryCount(message) < retryTemplate.getMaxRetryCount()) {
            incrementRetryCount(message);

            //Set the expiration of the retry message
            message.getMessageProperties().setExpiration(String.valueOf(retryTemplate.getNextRetryInterval(getRetryCount(message)).longValue()));

            RepublishMessageRecoverer retryRecoverer = null;
            if(getRetryCount(message) != null && getRetryCount(message) > 0) {
                retryRecoverer = retryExecutors.get(getRetryCount(message)-1);
            } else {
                retryRecoverer = retryExecutors.get(0);
            }
            retryRecoverer.recover(message, cause);
        } else {
            //Retries exchausted - do nothing
        }
    }

    private void createRetryExecutor(AmqpTemplate template, String exchangeName) {
        RepublishMessageRecoverer retryRecoverer = new RepublishMessageRecoverer(template, exchangeName);
        retryRecoverer.errorRoutingKeyPrefix(""); //Set KeyPrefix to "" so original key is reused during retries
        retryExecutors.add(retryRecoverer);
    }   

    private Integer getRetryCount(Message msg) {
        Integer retryCount;
        if(msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME) == null) {
            retryCount = 1;
        } else {
            retryCount =  (Integer) msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME);
        }

        return retryCount;
    }

    private void incrementRetryCount(Message msg) {
        Integer retryCount;
        if(msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME) == null) {
            retryCount = 1;
        } else {
            retryCount =  (Integer) msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME)+1;
        }
        msg.getMessageProperties().getHeaders().put(RETRY_COUNT_HEADER_NAME, retryCount);
    }

}

此处未发布'CustomRetryTemplate'的代码，但它包含maxRetryCount，initialInterval和multiplier的简单变量。

答案 2 :(得分：0)

您是否看过rabbitmq delayer插件，它会延迟交换机上的消息而不是队列？根据文档，发送到延迟交换的消息似乎在交换级别持久。

使用自定义重试计数邮件标题＆amp;在delayer交换中，我们可以实现非阻塞行为而不会出现这些中间队列的丑陋，dlx＆amp;模板组合

https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/

相关问题
最新问题