异步注释和并行处理

Question

这里首先是环境：

• 将大量消息发布到队列的服务。这是一个运行在容器中的Spring Boot应用程序，我可以访问它（意味着可以修改它）
• 另一个服务使用此队列，处理消息以创建xml报告，并将此报告发送到其他API端点。这也是在容器中运行的Spring Boot应用程序，我也可以访问它。
• 有时端点响应速度非常慢。我不了解端点的内部系统，也无法访问它。

我的消费者就是这样：

@Component
public class MyConsumer {

   private MyReportCreator reportCreator;
   private MyReportSender reportSender;

   @Autowired
   public MyConsumer(MyReportCreator reportCreator, MyReportSender reportSender) {
      this.reportCreator = reportCreator;
      this.reportSender = reportSender;
   }

   public void consume(MyMessage message) {
      try {
         String report = reportCreator.create(message);
         reportSender.send(report);
      } catch(Exception e) {
         logger.error("Error occured while reporting", e);
      }
   } 
}

问题是；使用者一一消耗队列，但是发布者每秒推送数千条消息，因此队列堆积得非常快。为了解决这个问题，我在应用程序中添加了@EnableAsync批注，在@Aysnc方法中添加了MyReportSender.send批注，并配置了一个TaskExecuter：

@Component
public class MyReportSender {

   private RestTemplate restTemplate;

   @Autowired
   public MyReportSender(RestTemplate restTemplate) {
      this.restTemplate = restTemplate;
   }

   @Async
   public void send(String report) {
      \\ sends the report using restTemplate.exchange, ignores result
   }

}

@SpringBootApplication
@EnableAsync
public class MyApp {

    public static void main(String[] args) {
        SpringApplication.run(MyApp.class, args);
    }

    @Bean   
    public Executor taskExecutor() {    
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); 
        executor.setCorePoolSize(100);  
        executor.setMaxPoolSize(100);   
        executor.setQueueCapacity(200); 
        executor.initialize();  
        return executor;    
    }
}

第二次发布此更改后，队列立即融化，但是一段时间后，我得到了这个异常：

Platform exception message: Executor [java.util.concurrent.ThreadPoolExecutor@1a9718ae[Running, pool size = 100, active threads = 100, queued tasks = 200, completed tasks = 3705]] did not accept task: org.springframework.aop.interceptor.AsyncExecutionInterceptor$$Lambda$584/29229306@7dd254ed

Platform exception trace: org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@1a9718ae[Running, pool size = 100, active threads = 100, queued tasks = 200, completed tasks = 3705]] did not accept task: org.springframework.aop.interceptor.AsyncExecutionInterceptor$$Lambda$584/29229306@7dd254ed
    at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:344)
    at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.doSubmit(AsyncExecutionAspectSupport.java:290)
    at org.springframework.aop.interceptor.AsyncExecutionInterceptor.invoke(AsyncExecutionInterceptor.java:129)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
    at com.personal.MyReportSender$$EnhancerBySpringCGLIB$$6f1f0df4.send(<generated>)
    at com.personal.MyReportSender.sendReport(MyReportSender.java:95)
    at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:246)
    at org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean.invoke(GenericScope.java:494)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
    at com.personal.MyReportSender$$EnhancerBySpringCGLIB$$604ed475.send(<generated>)
    at com.personal.MyConsumer.consume(MyConsumer.java:50)
    at com.personal.CommonMessageProcessor.process(CommonMessageProcessor.java:44)
    at com.personal.CommonMessageReceiver.receiveMessage(CommonMessageReceiver.java:52)
    at com.personal.CommonMessageReceiver.receiveMessage(CommonMessageReceiver.java:43)
    at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:280)
    at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:363)
    at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:292)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1547)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1473)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1461)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1456)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1405)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:870)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:854)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:78)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1137)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1043)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3cc54c9a rejected from java.util.concurrent.ThreadPoolExecutor@1a9718ae[Running, pool size = 100, active threads = 100, queued tasks = 200, completed tasks = 3705]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
    at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:341)
    ... 37 more

因此，当任务执行器正在处理100条消息且其内部队列中有200条消息并获得新消息时，它拒绝该消息。不好。

我想要的基本上是一次消耗该队列，例如100条消息，而不必是这种@Async解决方案。任何想法，评论，帮助都值得赞赏...