Question

我的kafka侦听器应按顺序处理消息，onMessage方法应同步处理消息，我不希望我的侦听器同时处理多个消息，onmessage方法首先停止

org.springframework.kafka.listener.MessageListenerContainer

然后将有效负载委托给同步方法，完成处理后，启动侦听器。 cousrse的其他选项是使用阻塞队列，执行服务等，需要更好的策略来实现这一目标，kafka使用者是否具有构建用于串行处理消息的功能？这是我的代码。

我将实现更改为此

    public static class KafkaReadMsgTask implements  Runnable{

    @Override
    public void run() {
        KakfaMsgConumerImpl  kakfaMsgConumerImpl=null;;
        try{
            kakfaMsgConumerImpl=SpContext.getBean(KakfaMsgConumerImpl.class);
            kakfaMsgConumerImpl.pollFormDef();
            kakfaMsgConumerImpl.pollFormData();
      } catch (Exception e){
          logger.error(" kafka listener  errors "+e);
          kakfaMsgConumerImpl.pauseTask();
       }  
    }
}


    @Component
public static class KakfaMsgConumerImpl {

    @Autowired
    ObjectMapper  mapper;

    @Autowired
    FormSink  formSink;

    @Autowired
    Environment  env;

    @Resource(name="formDefConsumer")
    Consumer formDefConsumer;

    @Resource(name="formDataConsumer")
    Consumer formDataConsumer;

    ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();

    public void startPolling() throws Exception{
        executor.scheduleAtFixedRate(new KafkaReadMsgTask(),10, 3,TimeUnit.SECONDS);
    }
    public void pauseTask(){
        try{
            Thread.sleep (120000l);
        }catch(Exception e){
            throw new RuntimeException(e);
        }
    }
    public void  pollFormDef() throws Exception{
        ConsumerRecords<Long, String> records =formDefConsumer.poll(0);
        if(!records.isEmpty()){
            int recordsCount=records.count();
            if(logger.isDebugEnabled()){
                logger.debug(" form-def consumer poll records size "+recordsCount);
            }
            if(records.count()>1){
                logger.warn(" form-def consumer poll returned records more than 1 , expected 1 , received "+recordsCount);
            }
            ConsumerRecord<Long,String> record= records.iterator().next();
            processFormDef(record.key(), record.value());
        }
    }
    void pollFormData() throws Exception{
        ConsumerRecords<Long, String> records =formDataConsumer.poll(0);
        if(!records.isEmpty()){
            int recordsCount=records.count();
            if(logger.isDebugEnabled()){
                logger.debug(" form-data consumer poll records size "+recordsCount);
            }
            if(records.count()>1){
                logger.warn(" form-data consumer poll returned records more than 1 , expected 1 , received "+recordsCount);
            }               ConsumerRecord<Long,String> record= records.iterator().next();
            processFormData(record.key(), record.value());
        }
    }
    void processFormDef(Long key, String msg) throws  Exception{
        if(logger.isDebugEnabled()){
            logger.debug(" key "+key+" payload : "+msg);
        }
        FormDefinition  formDefinition= mapper.readValue(msg, FormDefinition.class);
        formSink.createFromDef(formDefinition);
        logger.debug(" processed  message,  key: "+key+ " msg : "+msg);
        Thread.sleep(60000l);
    }

    void processFormData(Long key, String msg) throws  Exception{
        if(logger.isDebugEnabled()){
            logger.debug(" key "+key+" payload : "+msg);
        }
        FormData  formData= mapper.readValue(msg, FormData.class);
        formSink.persists(formData);
        logger.debug(" processed  message,  key: "+key+ " msg : "+msg);
        Thread.sleep(60000l);
    }



}

Answer 1

使用消息驱动的侦听器容器不是此应用程序的正确技术；看来您想交替使用来自两个不同主题的消息。

此外，在使用者线程上停止容器无论如何都不会生效，直到该线程退出该方法，此时使用者将被关闭。

我建议您使用消费者工厂来创建两个消费者。订阅主题，将每个主题上的max.poll.records设置为1，并交替调用每个主题上的poll()方法。

kafka使用者关闭运行时，顺序处理消息

1 个答案: