大家好,
更早之前,在生产中,我们只有一个带有Broker.id = 0的KAFKA(和Zookeeper)实例,服务器是PRD-1。一切都很好,直到我们将其转换为群集以恢复弹性。
因此,在将kafka的单个工作实例转换为具有2个节点的集群时,我们在$ KAFKA_HOME / config / server.properties中进行了以下更改 我们还在server.1和server.2属性的zookeeper配置中做了类似的更改。
PRD-1:
port=9092
broker.id=1 #Note change in broker id here. This was a mistake.
advertised.host.name=PRD-1.KPT.COM
zookeeper.connect=PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181
PRD-2:
port=9092
broker.id=2
advertised.host.name=PRD-2.KPT.COM
zookeeper.connect=PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181
此更改之后,我们启动了集群,它从代理1和2开始。它使用来自多个生产者的消息。一切正常,群集已启动并运行。
但是 没有任何使用者能够连接到新创建的kafka群集以使用消息。当我检查存储所有使用者偏移量的内部主题(主题:__consumer_offsets)时,我发现它是在将实例转换为集群之前创建的。我们发现了以下奇怪的细节:
./kafka-topics.sh --describe --zookeeper PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 0 Isr: 0
很显然,内部主题__consumer_offsets仍配置有broker.id = 0(当PRD-1在没有群集的单台计算机上运行时是否为PRD-1)。
鉴于上述情况,我尝试使用以下命令进行重新分配。
./kafka-topics.sh --describe --zookeeper PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 1,2,0 Isr: 0
但是现在重新分配使情况变得更糟,我有3个副本(0,1,2),其中0是不再存在的经纪人。
现在我的主题__consumer_offsets的复制因子为3,因为我在群集中只有2个节点(代理1和2)。
主题重新分配现在失败,并显示以下错误消息。
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-22
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-30
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-8
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-21
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-4
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-27
.....
..
..
Reassignment of partition __consumer_offsets-22 failed
Reassignment of partition __consumer_offsets-30 failed
Reassignment of partition __consumer_offsets-8 failed
有人可以帮忙做什么,以便主题__consumer_offsets可以识别群集并相应地更新其详细信息?。
或
我需要删除主题__consumer_offsets并弹跳kafka,以便它自动创建它?。
任何寻求帮助/解决方案/方法的指针都将受到高度赞赏。
谢谢。