使用者仅从具有复制因子3和分区1的3代理配置中读取特定的代理。

时间：2019-03-14 15:54:34

标签： apache-kafka

场景

我已经使用https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz

中的shell脚本在一台机器上启动了用于多代理配置的所有组件。
  1. 使用zookeeper-properties启动Zookeeper
  2. 启动了具有3个不同服务器属性的3个代理。它们仅在这些配置值上有所区别
broker.id
log.dirs
port

我也尝试过更改offsets.topic.replication.factor和transaction.state.log.replication.factor，但我不认为它们是相关的。

注意：我启动经纪人的顺序是0，1，2

  1. 创建一个具有复制因子3和一个分区的主题
bin/kafka-topics.sh --create --topic repl_topic --zookeeper localhost:2181 --replication-factor 3 --partitions 1
  1. 开始成为控制台生产商和消费者
bin/kafka-console-producer.sh --topic repl_topic --broker-list localhost:9092,localhost:9093,localhost:9094
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 -topic repl_topic --from-beginning

生产者和消费者似乎正常工作。但是，如果我通过Ctrl-C关闭代理0（即使不是领导者），则使用者将收到警告，但不再接收来自生产者的消息。仅当代理0重新启动时，消费者才会收到所有消息。

我的结论

使用者仅依赖于代理0。它不会与其他人互动。

我的问题

为什么？

1 个答案:

答案 0 :(得分：-1)

我终于找到了问题，并已解决。 检查__consumer_offsets，我发现它没有被复制。 

 bin/kafka-topics.sh --topic __consumer_offsets --zookeeper localhost:2181 --describe
Topic:__consumer_offsets    PartitionCount:50   ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets   Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 2    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 3    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 4    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 5    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 6    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 7    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 8    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 9    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 10   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 11   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 12   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 13   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 14   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 15   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 16   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 17   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 18   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 19   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 20   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 21   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 22   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 23   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 24   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 25   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 26   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 27   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 28   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 29   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 30   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 31   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 32   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 33   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 34   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 35   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 36   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 37   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 38   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 39   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 40   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 41   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 42   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 43   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 44   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 45   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 46   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 47   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 48   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 49   Leader: 0   Replicas: 0 Isr: 0

的确，我第一次启动使用者时，它是针对一个Replication_factor等于1的主题。那时，使用者仅创建了一个副本。该主题不再更新，因此如果代理0下降，其他代理将无法看到该主题。

解决方案

可以通过以下命令为每个分区建立新的复制级别 bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file new_reassignment.json --execute

在那里使用的 new_reassignment.json json文件具有以下内容

{"version":1,"partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":3,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":4,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":5,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":6,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":7,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":8,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":9,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":10,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":11,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":12,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":13,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":14,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":15,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":16,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":17,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":18,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":19,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":20,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":21,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":22,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":23,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":24,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":25,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":26,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":27,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":28,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":29,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":30,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":31,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":32,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":33,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":34,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":35,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":36,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":37,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":38,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":39,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":40,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":41,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":42,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":43,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":44,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":45,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":46,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":47,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":48,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":49,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

此时，可以成功管理任何代理的失败。

__ consumer_offsets-额外信息

此特定主题中的分区数默认设置为50。 消费者触发其创建。消费者主要使用该主题来提交每个主题：分区的接收消息。如果此__consumer_offsets主题对其他代理不可见，则代理将无法通知消费者使用新消息。

相关问题
最新问题