消费者之间的主题共享

时间:2020-09-08 19:47:28

标签: spring-boot apache-kafka spring-kafka

我正在开发一个Springboot应用程序,它在Kafka主题(例如700个主题,每个主题有10个分区,即7000个分区)中重新平衡。但我想启动多个dockerized应用程序实例,其中该应用程序将包含所有700个主题名称,但它应仅选择前25个分区,然后取消选择其他分区)

@KafkaListener( topics = "#{kafkaProperties.getTopics()}" )

kafkaProperties.getTopics()返回所有700个主题名称

1 个答案:

答案 0 :(得分:0)

正如我在最初的评论中要求澄清的那样,对于这样的分区分配方案,您将需要实现自定义useEffect(() => { const onAnswerCall = () => { console.log('=== onAnswerCall ::: ===', calls); }; RNCallKeep.addEventListener('answerCall', onAnswerCall); return () => { RNCallKeep.removeEventListener('answerCall', onAnswerCall); }; }, [calls]);

每个实例应选择数量有限的线程,每个线程具有一个主题/分区

实例不“挑选”主题/分区;选择一个实例,并确定哪个实例获得哪个主题/分区。

请参阅其javadocs。

ConsumerPartitionAssignor

要将更多线程(消费者)添加到实例,请增加容器并发性。

对于大量实例,我建议使用/** * This interface is used to define custom partition assignment for use in * {@link org.apache.kafka.clients.consumer.KafkaConsumer}. Members of the consumer group subscribe * to the topics they are interested in and forward their subscriptions to a Kafka broker serving * as the group coordinator. The coordinator selects one member to perform the group assignment and * propagates the subscriptions of all members to it. Then {@link #assign(Cluster, GroupSubscription)} is called * to perform the assignment and the results are forwarded back to each respective members * * In some cases, it is useful to forward additional metadata to the assignor in order to make * assignment decisions. For this, you can override {@link #subscriptionUserData(Set)} and provide custom * userData in the returned Subscription. For example, to have a rack-aware assignor, an implementation * can use this user data to forward the rackId belonging to each member. */ public interface ConsumerPartitionAssignor { 重新平衡协议。