高级Kafka消费者只消耗生产者发送的一半消息

时间:2017-02-02 15:10:03

标签: apache-kafka apache-zookeeper kafka-consumer-api

我已根据示例页面实施了高级消费者:https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example

当代码运行时,它只消耗一半的消息。我有一个基本的3节点zookeeper集群和2个kafka经纪人。当我运行简单的消费者代码(不是高级消费者)时,将消耗所有消息。任何想法将不胜感激。

消费者代码

   public void run() {
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put("test", 2);
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        List<KafkaStream<byte[], byte[]>> streams = consumerMap.get("test");

        executor = Executors.newFixedThreadPool(2);

        int threadNumber = 0;

        for (final KafkaStream stream : streams) {

            executor.submit(new Consumer(stream, threadNumber));
            threadNumber++;
        }
    }

    private static ConsumerConfig createConsumerConfig() {
        Properties props = new Properties();
        props.put("zookeeper.connect", "zookeeper01:2181,zookeeper02:2181,zookeeper03:2181");
        props.put("group.id", "Consumers");
        props.put("zookeeper.session.timeout.ms", "10000");
        props.put("enable.auto.commit", "true");
        props.put("zookeeper.sync.time.ms", "1000");
        props.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
        props.put("auto.commit.interval.ms", "500");

        return new ConsumerConfig(props);
    }

0 个答案:

没有答案