Question

我正在尝试构建一个涉及两个主题和一个消费者的kafka服务。每个主题都有10个分区。目前，我有一个设置，供消费者听两个主题。

我想使处理优先级为基础。我想优先考虑主题1中的事件而不是主题2中的事件。

因此，目标是仅在主题1中没有事件时才处理主题2中的事件。我一直在尝试使用Consumer.Pause选项，但到目前为止我还没有运气，因为我是Kafka的初学者。

欣赏一个代码片段，它可以显示实现此目的所需的内容。

Answer 1

这是一种解决方案：

@SpringBootApplication
public class So63855196Application {

    public static void main(String[] args) {
        SpringApplication.run(So63855196Application.class, args);
    }

    @Bean
    public NewTopic hiT() {
        return TopicBuilder.name("hiT").partitions(1).replicas(1).build();
    }

    @Bean
    public NewTopic loT() {
        return TopicBuilder.name("loT").partitions(1).replicas(1).build();
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0, 10).forEach(i -> template.send("loT", "foo" + i));
            Thread.sleep(3000);
            IntStream.range(0,  2).forEach(i -> template.send("hiT", "bar" + i));
        };
    }

}

@Component
class Listener {

    private static final Logger log = LoggerFactory.getLogger(Listener.class);

    private final KafkaListenerEndpointRegistry registry;

    private final AtomicBoolean paused = new AtomicBoolean();

    Listener(KafkaListenerEndpointRegistry registry) {
        this.registry = registry;
    }

    @KafkaListener(id = "high", topics = "hiT")
    @KafkaListener(id = "low", topics = "loT",
            properties = ConsumerConfig.MAX_POLL_RECORDS_CONFIG + ":1")
    public void listen(String in, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) throws InterruptedException {
        System.out.println(in + " from " + topic);
        if (topic.equals("hiT")) {
            if (this.paused.compareAndSet(false, true)) {
                log.info("Hi-pri active; pausing low-pri");
                this.registry.getListenerContainer("low").pause();
            }
        }
        Thread.sleep(2000);
    }

    @EventListener
    public void event(ListenerContainerIdleEvent event) {
        if (event.getListenerId().startsWith("high")) {
            if (this.paused.compareAndSet(true, false)) {
                log.info("Hi-pri idle; resuming low-pri");
                this.registry.getListenerContainer("low").resume();
            }
        }
    }

}

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.idle-event-interval=1000

和

foo0 from loT
foo1 from loT
bar0 from hiT
2020-09-14 09:40:28.630  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri active; pausing low-pri
bar1 from hiT
2020-09-14 09:40:37.647  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri idle; resuming low-pri
foo2 from loT
foo3 from loT
foo4 from loT
foo5 from loT
foo6 from loT
foo7 from loT
foo8 from loT
foo9 from loT

这仅在容器具有concurrency=1时有效；当容器具有多个线程时，它将需要更多的技巧。

Springboot KafkaListener侦听多个主题

1 个答案: