Question

我们一直在努力修复不想使用数据的消费者。它每4秒轮询一次，通常只收到大约100条记录。最大它拉大约18000.然后它将原始字节写入文件。所有快速操作。

但消费者群组电话会显示以下内容

[xyz@AZ254PP01 kafka]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --new-consumer --group postproc-red2
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
postproc-red2, targeting-seg, 0, 21921226, 21921388, 162, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 0, 72891022, 78349037, 5458015, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 0, 16665679, 18122272, 1456593, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 0, 50847858, 56468186, 5620328, consumer-2_/192.168.201.121
postproc-red2, targeting-seg, 1, 20733636, 24858954, 4125318, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 1, 75084847, 89556283, 14471436, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 1, 19648100, 19648100, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 1, 66549015, 66549015, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-seg, 2, 24903106, 28041684, 3138578, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 2, 99480656, 99480656, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 2, 19982077, 24848188, 4866111, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 2, 69611742, 76253273, 6641531, consumer-2_/192.168.201.121

当它记录时，它似乎只想从targets-seg-0消费，但一次只能消耗60-90条记录。它有时会同时从所有0个分区中拉出几千个。这些数据无论如何都没有意义，当涉及到寻找问题的其他地方时，我们处于亏损状态。它只是一个消费者。消耗这些主题的所有分区的一个过程。

过去两天的制作人似乎只写0分区。我们也不知道原因。

我试着看看它是否适用于所有未被拉出超过60秒的分区上的toToEnd。

14:58:33.896 [Kafka-0] INFO  c.t.e.p.s.red2.ConsumerService - Processing records 3263 of Topics [targeting-seg-0, targeting-visit-0, targeting-vis-0, targeting-vc-0]
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vc-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vc-2
14:58:38.005 [Kafka-0] INFO  c.t.e.p.s.red2.ConsumerService - Processing records 3284 of Topics [targeting-seg-0, targeting-visit-0, targeting-vis-0, targeting-vc-0]
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-2
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-2
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-2

代码是订阅（主题）。轮询（500）。 seekToEnd。然后我希望在下次民意调查中，seekToEnd会修复一个有问题的offsetcommit。

任何想法Kafka设置的问题可能是什么或如何解决它。它几乎在默认设置下运行。没有分区具有副本 - ＆gt;复制因子1.我们打算切换到复制因子3，因为生产者是三台服务器。

Answer 1

生产者根据消息的键的哈希值选择目标分区＆＃34;生成＆＃34;。您应该检查过去两天的消息是否始终具有相同的密钥。

在消费者方面，您确定postproc-red2仅用于此消费者吗？您确定在同一组中拥有唯一的消费者ID吗？ Hovewer最佳做法是使消费者数量等于源主题的分区号。

希望这可以提供帮助

卡夫卡抵消滞后远远落后于原因？

1 个答案: