Kafka:主题中奇怪的分区分配

时间:2017-05-12 16:15:09

标签: apache-kafka kafka-consumer-api spring-kafka

我不确定如何解释我与卡夫卡面临的问题,但我会尽我所能。我在同一个消费者群体中有一组4个消费者:

  

absolutegrounds.helper.processor

从具有5个分区的主题消费;因此,该组中的每个消费者被分配到1个分区,1个消费者被分配到2个分区,以便公平地在4个消费者之间分配5个分区。

但由于某种原因我无法弄清楚,初始分配变成只有2个消费者被分配到所有可用分区,即1个消费者有3个分区,1个消费者有2个分区。然而,理论上仍有4位消费者属于同一消费群体:

[medinen@ocvlp-rks001 kafka_2.11-0.10.0.1]$ ./bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --new-consumer --bootstrap-server localhost:9092 --describe --group absolutegrounds.helper.processor
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
absolutegrounds.helper.processor AG_TASK_SOURCE                 0          27286           31535           4249            consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE                 1          28015           28045           30              consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE                 2          35437           40091           4654            consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE                 3          31765           31874           109             consumer-1_/10.132.8.23
absolutegrounds.helper.processor AG_TASK_SOURCE                 4          33279           38003           4724            consumer-1_/10.132.8.23

最离奇的行为是其他2个消费者群体(根据上面Kafka的回复)似乎仍然根据我在我的应用程序中看到的日志消耗了这个主题,尽管我找不到它们作为消费者群体的一部分。更奇怪的是,应该将1个消费者分配给主题中的所有分区,而另一个仅分配给分区4.请参阅应用程序中的日志(这是使用Spring Kafka的Spring Boot应用程序):< / p>

第一个左消费者:

-  - 08/05/2017 12:27:29.119 - [-kafka-consumer-1] INFO  o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions [AG_TASK_SOURCE-0, AG_TASK_SOURCE-1, AG_TASK_SOURCE-2, AG_TASK_SOURCE-3, AG_TASK_SOURCE-4] for group absolutegrounds.helper.processor

第二位消费者:

 -  - 08/05/2017 12:27:19.044 - [-kafka-consumer-1] INFO  o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions [AG_TASK_SOURCE-4] for group absolutegrounds.helper.processor

试图理解这种行为背后的原因,我已经研究了为消费者存储所有偏移的主题:

  

__ consumer_offsets

使用此命令:

kafka/kafka_2.11-0.10.0.1/bin/kafka-console-consumer.sh --consumer.config /tmp/consumer.config --formatter "kafka.coordinator.GroupMetadataManager\$GroupMetadataMessageFormatter" --zookeeper ocvlp-rks003:2181 --topic __consumer_offsets --from-beginning | grep "absolutegrounds.helper.processor"

这就是我发现的:

absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb -> [consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb,consumer-1,/10.132.8.23,10000], consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332 -> [consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332,consumer-1,/10.132.9.128,10000], consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf -> [consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf,consumer-1,/10.132.8.22,10000], consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7 -> [consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7,consumer-1,/10.132.9.129,10000])]

absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-66de4a46-538c-425f-8e95-5a00ff5eb5fd -> [consumer-1-66de4a46-538c-425f-8e95-5a00ff5eb5fd,consumer-1,/10.132.9.129,10000])]

absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6 -> [consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6,consumer-1,/10.132.9.128,10000], consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21 -> [consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21,consumer-1,/10.132.8.23,10000])]

absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6 -> [consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6,consumer-1,/10.132.9.128,10000], consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21 -> [consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21,consumer-1,/10.132.8.23,10000])]

根据Kafka的回应,我可以看到在某个时间点所有4个消费者在分区之间正确分配:

absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb -> [consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb,consumer-1,/10.132.8.23,10000], consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332 -> [consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332,consumer-1,/10.132.9.128,10000], consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf -> [consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf,consumer-1,/10.132.8.22,10000], consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7 -> [consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7,consumer-1,/10.132.9.129,10000])]

但是,稍后,分配更改为当前方案,其中消费者组中的4个消费者中只有2个被分配了分区。

我很难理解可能导致这种情况的原因,但我找不到有效的答案来弄明白并解决它。

任何人都可以在这里帮忙吗?感谢。

0 个答案:

没有答案