我有一个由3个VM组成的kafka zookeeper集群。我的应用程序在2节点集群中运行。当我开始在VM 1中运行我的消费者时，我看到我的消费者试图声称分区并成功。
2016-03-31 14:29:38,154 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_vm-7666-2q234324], Consumer MYGROUP_VM-7666-2q234324 rebalancing the following partitions: ArrayBuffer(0, 1, 2, 3) for topic MY_TOPIC with consumers: List(MYGROUP_VM-7666-2q234324-0, MYGROUP_VM-7666-2q234324-1)
2016-03-31 14:29:38,154 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-0 attempting to claim partition 0
2016-03-31 14:29:38,156 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-0 attempting to claim partition 1
2016-03-31 14:29:38,159 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-1 attempting to claim partition 2
2016-03-31 14:29:38,162 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-1 attempting to claim partition 3
2016-03-31 14:29:38,168 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-0 successfully owned partition 0 for topic MY_TOPIC
2016-03-31 14:29:38,171 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-1 successfully owned partition 3 for topic MY_TOPIC
2016-03-31 14:29:38,173 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-1 successfully owned partition 2 for topic MY_TOPIC
2016-03-31 14:29:38,176 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], MYGROUP_VM-7666-2q234324-0 successfully owned partition 1 for topic MY_TOPIC
2016-03-31 14:29:38,177 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], Updating the cache
2016-03-31 14:29:38,181 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324], Consumer MYGROUP_VM-7666-2q234324 selected partitions : MY_TOPIC:0: fetched offset = 875: consumed offset = 875,MY_TOPIC:1: fetched offset = 868: consumed offset = 868,MY_TOPIC:2: fetched offset = 821: consumed offset = 821,MY_TOPIC:3: fetched offset = 873: consumed offset = 873
2016-03-31 14:29:38,185 - INFO (kafka.utils.Logging$class:info:?) - [MYGROUP_VM-7666-2q234324-leader-finder-thread], Starting`
什么时候启动我的其他VM，我得到以下日志。我的第二个消费者试图为同一个主题声明一些分区3次，并且失败可怕。
15:19:00,109 INFO ConsumerFetcherManager:info(?) | [ConsumerFetcherManager-1459549139260] Stopping leader finder thread
15:19:00,110 INFO ConsumerFetcherManager:info(?) | [ConsumerFetcherManager-1459549139260] Stopping all fetchers
15:19:00,112 INFO ConsumerFetcherManager:info(?) | [ConsumerFetcherManager-1459549139260] All connections stopped
15:19:00,114 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], Cleared all relevant queues for this fetcher
15:19:00,116 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], Cleared the data chunks in all the consumer message iterators
15:19:00,117 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], Committing all offsets after clearing the fetcher queues
15:19:00,119 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], Releasing partition ownership
15:19:00,123 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], Consumer MYGROUP_VM_32432qrew rebalancing the following partitions: ArrayBuffer(0, 1, 2, 3) for topic MY_TOPIC with consumers: List(MYGROUP_VM_32432qrew-0, MYGROUP_VM-7666-2q234324-0, MYGROUP_VM-7666-2q234324-1)
15:19:00,126 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], MYGROUP_VM_32432qrew-0 attempting to claim partition 0
15:19:00,158 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], MYGROUP_VM_32432qrew-0 attempting to claim partition 1
15:19:00,247 INFO ZkUtils$:info(?) | conflict in /consumers/ALMGROUP/owners/MY_TOPIC/0 data: MYGROUP_VM_32432qrew-0 stored data: ALMGROUP_almjobs-7666-1459460205521-f8c2fb3d-0
15:19:00,249 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], waiting for the partition ownership to be deleted: 0
15:19:00,307 INFO ZkUtils$:info(?) | conflict in /consumers/ALMGROUP/owners/MY_TOPIC/1 data: MYGROUP_VM_32432qrew-0 stored data: ALMGROUP_almjobs-7666-1459460205521-f8c2fb3d-0
15:19:00,308 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], waiting for the partition ownership to be deleted: 1
15:19:00,310 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], end rebalancing consumer MYGROUP_VM_32432qrew try #0
15:19:00,311 INFO ZookeeperConsumerConnector:info(?) | [MYGROUP_VM_32432qrew], Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered
15:19:00,312 INFO ConsumerFetcherManager:info(?) | [ConsumerFetcherManager-1459549139260] Stopping leader finder thread`
我的配置 -
kafka.required.acks=1
kafka.producer.type=async
kafka.serializer.class=kafka.serializer.StringEncoder
kafka.zk.connect=vm1:8080,vm2:8080,vm3:8080
kafka.zk.group.id=MYGROUP
kafka.zk.session.timeout.ms=100000
kafka.zk.connection.timeout.ms=100000
kafka.zk.sync.time.ms=1000
kafka.zk.commit.interval.ms=1000
queue.consumer.timeout.ms=1000
env.ssl.enabled=true
kafka.required=false
kafka.topic.partitions=1
kafka.topic.replicas=2
我错过了什么吗？我花了相当多的时间，但仍然面临同样的问题。我重新启动了vms，重新安装了zookeeper和kafka。我仍然不确定让重新平衡成功的缺失。