Question

我有3个节点（nodes0，node1，node2）Kafka集群（broker0，broker1，broker2），复制因子2和Zookeeper（使用与Kafka tar打包的zookeeper）在不同的节点（节点4）上运行。

我在启动zookeper然后剩下的节点后启动了代理0。在代理0日志中可以看到它正在读取__consumer_offsets并且它们似乎存储在代理0上。以下是样本日志：

Kafka版本： kafka_2.10-0.10.2.0

    2017-06-30 10:50:47,381] INFO [GroupCoordinator 0]: Loading group metadata for console-consumer-85124 with generation 2 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 10:50:47,382] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from __consumer_offsets-41 in 23 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,382] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from __consumer_offsets-44 (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,387] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from __consumer_offsets-44 in 5 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,387] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from __consumer_offsets-47 (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,398] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from __consumer_offsets-47 in 11 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,398] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from __consumer_offsets-1 (kafka.coordinator.GroupMetadataManager)

此外，我可以在同一代理0日志中看到GroupCoordinator消息。

[2017-06-30 14:35:22,874] INFO [GroupCoordinator 0]: Preparing to restabilize group console-consumer-34472 with old generation 1 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:22,877] INFO [GroupCoordinator 0]: Group console-consumer-34472 with generation 2 is now empty (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:25,946] INFO [GroupCoordinator 0]: Preparing to restabilize group console-consumer-6612 with old generation 1 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:25,946] INFO [GroupCoordinator 0]: Group console-consumer-6612 with generation 2 is now empty (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:38,326] INFO [GroupCoordinator 0]: Preparing to restabilize group console-consumer-30165 with old generation 1 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:38,326] INFO [GroupCoordinator 0]: Group console-consumer-30165 with generation 2 is now empty (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:43:15,656] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 14:53:15,653] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)

在使用 kafka-console-consumer.sh和kafka-console-producer.sh 测试群集的容错时，我看到了关于杀死经纪人1或经纪人2，消费者仍然可以收到来自生产者的新消息。重新平衡正确发生。

然而，杀死经纪人0导致任何数量的消费者都没有新的或旧的消息消费。下面是代理0被杀死之前和之后的主题状态。

之前

Topic:test-topic    PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: test-topic   Partition: 0    Leader: 2   Replicas: 2,0   Isr: 0,2
    Topic: test-topic   Partition: 1    Leader: 0   Replicas: 0,1   Isr: 0,1
    Topic: test-topic   Partition: 2    Leader: 1   Replicas: 1,2   Isr: 1,2

后

Topic:test-topic    PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: test-topic   Partition: 0    Leader: 2   Replicas: 2,0   Isr: 2
    Topic: test-topic   Partition: 1    Leader: 1   Replicas: 0,1   Isr: 1
    Topic: test-topic   Partition: 2    Leader: 1   Replicas: 1,2   Isr: 1,2

以下是代理0被杀后在消费者日志中看到的WARN消息

[2017-06-30 14:19:17,155] WARN Auto-commit of offsets {test-topic-2=OffsetAndMetadata{offset=4, metadata=''}, test-topic-0=OffsetAndMetadata{offset=5, metadata=''}, test-topic-1=OffsetAndMetadata{offset=4, metadata=''}} failed for group console-consumer-34472: Offset commit failed with a retriable exception. You should retry committing offsets. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-06-30 14:19:10,542] WARN Auto-commit of offsets {test-topic-2=OffsetAndMetadata{offset=4, metadata=''}, test-topic-0=OffsetAndMetadata{offset=5, metadata=''}, test-topic-1=OffsetAndMetadata{offset=4, metadata=''}} failed for group console-consumer-30165: Offset commit failed with a retriable exception. You should retry committing offsets. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

经纪人属性。其余的默认属性保持不变。

broker.id=0
delete.topic.enable=true

auto.create.topics.enable=false
listeners=PLAINTEXT://XXX:9092
advertised.listeners=PLAINTEXT://XXX:9092
log.dirs=/tmp/kafka-logs-test1
num.partitions=3
zookeeper.connect=XXX:2181

制作人资源。其余的默认属性保持不变。

bootstrap.servers=XXX,XXX,XXX
compression.type=snappy

消费者资产。其余的默认属性保持不变。

zookeeper.connect=XXX:2181
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group

据我所知，如果节点持有/代理GroupCoordinator和__consumer_offsets死亡，那么尽管新的领导者被选为分区，但消费者无法恢复正常运营。

我在post中看到了类似的内容。这篇文章建议重启死代理节点。但是，在生产环境中重新启动代理0之前，尽管有更多节点，但消息消耗会有延迟。

Q1：如何减轻上述情况？

Q2：有没有办法将GroupCoordinator，__ consumer_offsets更改为另一个节点？

感谢任何建议/帮助。

Answer 1

检查__consumer_offsets主题上的复制因子。如果它不是3则那就是你的问题。

运行以下命令kafka-topics --zookeeper localhost:2181 --describe --topic __consumer_offsets并查看输出的第一行是否显示“ReplicationFactor：1”或“ReplicationFactor：3”。

在首次设置一个节点进行试验时，这是一个常见问题，然后在复制因子为1的情况下创建此主题。稍后当您扩展到3个节点时，您忘记更改此现有主题的主题级别设置，以便即使您正在制作和使用的主题是容错的，偏移主题仍然只停留在代理0上。

使用__consumer_offsets杀死节点会导致消费者消息消失

1 个答案: