使用春季kafka中的注释为每个主题分别设置Kafka监听器

时间:2019-11-15 09:25:09

标签: spring spring-boot kotlin apache-kafka spring-kafka

我的应用程序正在侦听多个kafka主题。现在,我的侦听器如下所示,属性文件中的my.topics包含逗号分隔的kafka主题列表

@KafkaListener(topics = ["#{'\${my.topics}'.split(',')}"], groupId = "my.group", containerFactory = "myKafkaFactory")
fun genericMessageListener(myRequest: MyRequest, ack: Acknowledgment) {
   //do Something with myRequest
    ack.acknowledge()
}

我的ConcurrentKafkaListenerContainerFactory

@Bean
fun charterRequestKafkaListenerfactory(): ConcurrentKafkaListenerContainerFactory<String, CharterRequest> {
    val factory = ConcurrentKafkaListenerContainerFactory<String, CharterRequest>()
    factory.consumerFactory = DefaultKafkaConsumerFactory(configProps(), StringDeserializer(), CharterRequestDeserializer())
    factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL
    return factory
}

有没有一种方法可以动态地为每个主题创建一个单独的使用者,以便当我在my.topics的列表中添加另一个主题时,spring会自动为该主题创建一个单独的使用者。

我现在面临的问题是,如果任何主题中的一条消息出了问题,其他主题中的消息也会受到影响。

在高水平上,我正在寻找类似的东西

@Bean
fun myKafkaFactory(): ConcurrentKafkaListenerContainerFactory<String, MyRequest> {
    val factory = ConcurrentKafkaListenerContainerFactory<String, MyRequest>()
    factory.consumerFactory = DefaultKafkaConsumerFactory(configProps(), StringDeserializer(), MyRequestDeserializer())
    factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL
    factory.isOneConsumerPerTopic(true)
    return factory
}

以便factory.isOneConsumerPerTopic(true)将确保为数组中的每个主题创建一个单独的使用者。

我确实经历过How to create separate Kafka listener for each topic dynamically in springboot?。我正在寻找更多“更清洁”的解决方案。 :)

1 个答案:

答案 0 :(得分:1)

您可以将自定义PartitionAssignor添加到Kafka使用者配置中。

将容器并发设置为(至少)主题数,并让分配器将每个主题的分区分配给特定使用者。

/**
 * This interface is used to define custom partition assignment for use in
 * {@link org.apache.kafka.clients.consumer.KafkaConsumer}. Members of the consumer group subscribe
 * to the topics they are interested in and forward their subscriptions to a Kafka broker serving
 * as the group coordinator. The coordinator selects one member to perform the group assignment and
 * propagates the subscriptions of all members to it. Then {@link #assign(Cluster, Map)} is called
 * to perform the assignment and the results are forwarded back to each respective members
 *
 * In some cases, it is useful to forward additional metadata to the assignor in order to make
 * assignment decisions. For this, you can override {@link #subscription(Set)} and provide custom
 * userData in the returned Subscription. For example, to have a rack-aware assignor, an implementation
 * can use this user data to forward the rackId belonging to each member.
 */
public interface PartitionAssignor {

您可以从AbstractPartitionAssignor开始。

请参见spring-kafka documentation

  

在听多个主题时,默认的分区分布可能不是您所期望的。例如...