Question

我们希望在一定间隔后（例如每5分钟）消耗一次记录。消费者属性是标准的：

@Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<Integer, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    factory.setConcurrency(1);
    factory.setBatchListener(true);
    factory.getContainerProperties().setPollTimeout(300000);
    factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.BATCH);
    return factory;
}

即使我更改了属性setPollTimeout，它也不会在定义的间隔（5分钟）后轮询，它会在30秒后继续轮询，这是我的日志：

2018-01-23 18:07:26.875 INFO 60905 --- [        2-0-C-1] c.t.k.s.consumer.FavoriteEventConsumer   : Consumed: san@1516710960000->1516711080000 2

2018-01-23 18:07:56.901 INFO 60905 --- [        2-0-C-1] c.t.k.s.consumer.FavoriteEventConsumer   : Consumed: san@1516710960000->1516711080000 4

我们尝试使用窗口化聚合构建一个kafka流应用程序，并计划在y间隔后使用窗口x。

我可以在课堂上看到：KafkaMessageListenerContainer，setConsumerTaskExecutor已设置：

if (containerProperties.getConsumerTaskExecutor() == null) {
        SimpleAsyncTaskExecutor consumerExecutor = new SimpleAsyncTaskExecutor(
                (getBeanName() == null ? "" : getBeanName()) + "-C-");
        containerProperties.setConsumerTaskExecutor(consumerExecutor);
    }

但是我们如何配置此（频率）线程池何时轮询记录。任何帮助表示赞赏。

Answer 1

您无法控制消费者轮询的比率，pollTimeout是poll()等待新记录到达的时间。如果新记录更频繁地到达，它将不会等待那么久。

如果您希望控制接收记录的速率，只需使用DefatulKafkaConsumerFactory创建消费者并随时轮询它。

你不能在@KafkaListener上使用它 - 你必须自己处理记录。

Answer 2

此功能在2.3版本中引入。

从2.3版开始，ContainerProperties提供了一个 idleBetweenPolls 选项可让主循环进入监听器容器在KafkaConsumer.poll（）调用之间休眠。实际的睡眠间隔从提供的选项和差异中选择最小值在max.poll.interval.ms使用者配置和当前记录批处理时间。

https://docs.spring.io/spring-kafka/reference/html/

KafkaListenerConfig.java

package br.com.sicredi.spi.icom.consumer.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import java.util.HashMap;
import java.util.Map;

@EnableKafka
@Configuration
public class KafkaListenerConfig {

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> concurrentKafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        
        factory.getContainerProperties().setIdleBetweenPolls(100); // 100 miliseconds
        
        return factory;
    }

    private ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfig());
    }

    private Map<String, Object> consumerConfig() {
        Map<String, Object> props = new HashMap<>();
        // ... 
        return props;
    }
}

Answer 3

如果要使用Spring @KafkaListener来控制使用Kafka的用户的速率，请按以下方式自动连接KafkaListenerEndpointRegistry bean，并访问所需的MessageListenerContainer。之后，您可以使用pause（）和resume（）功能来控制所需的行为。

@Autowired
private KafkaListenerEndpointRegistry listener;

@Autowired
private Map<String, Set<String>> getTopicListenerMap(){
    List<String> ids = new ArrayList<>(listener.getListenerContainerIds());
    Map<String, Set<String>> topicListenerMap = new HashMap<>();
    for(String topic: topics){
        topicListenerMap.put(topic, new HashSet<>());
    }
    for(String key: ids){
        for (String topic : listener.getListenerContainer(key).getContainerProperties().getTopics()){
            topicListenerMap.get(topic).add(key);
        }
    }
    return topicListenerMap;
}

@KafkaListener(topics = "topic", containerFactory = "smsListener")
public void listenWithHeaders(@Payload List<String> messageList, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitionList,
                              @Header(KafkaHeaders.OFFSET) List<Integer> offsetList) {
    try{
        LOG.info("Received message count: "+(messageList!=null ? messageList.size(): 0)+", offset start: "+offsetList.get(0)+", end: "+offsetList.get(offsetList.size()-1));
        pauseIfRequired(topic);
        for(int i=0; i<messageList.size(); i++){
            // process the messages
        }
    }catch (Exception e){
        LOG.error("", e);
    }finally {
        resumeIfPaused(topic);
    }
}

private void pauseIfRequired(String topic){
    try{ 
        boolean flag = pausingCondition;
        if(flag){
            LOG.info("pausing topic: "+topic);
            for(String listenerKey: getTopicListenerMap().get(topic)){
                listener.getListenerContainer(listenerKey).pause();
            }
            LOG.info("topic paused: "+topic);
        }
    } catch (Exception e){
        LOG.error("", e);
    }
}

private void resumeIfPaused(String topic){
    try {
        for (String listenerKey : getTopicListenerMap().get(topic)) {
            LOG.info("topic: "+topic+", containerPauseRequested: "+listener.getListenerContainer(listenerKey).isPauseRequested());
            if (listener.getListenerContainer(listenerKey).isPauseRequested()) {
                LOG.info("waiting to resume topic: " + topic + ", listener key: " + listenerKey);
                // wait while the condition to resume is fulfilled
                LOG.info("resuming topic: " + topic + ", listener key: " + listenerKey);
                listener.getListenerContainer(listenerKey).resume();
                LOG.info("topic resumed: " + topic + ", listener key: " + listenerKey);
            }
        }
    } catch (Exception e){
        LOG.error("", e);
    }
}

Spring @KafkaListener在一定的时间间隔后执行并轮询记录

3 个答案: