Question

您好我实际上是在尝试实现高级消费者代码而不是自动提交我想使用手动提交。我的主要目的是当我提交偏移量并再次重新启动消费者时它不显示旧的和读取消息。我实现了下面的代码.Plz帮助我。

    public VulabKafkaConsumer(String a_zookeeper, String a_groupId, String a_topic) {
        consumer = kafka.consumer.Consumer.createJavaConsumerConnector(
                createConsumerConfig(a_zookeeper, a_groupId));
        this.topic = a_topic;
    }

    public void run() {
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(topic, new Integer(1));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
        for (final KafkaStream stream : streams) {
            ConsumerIterator<byte[], byte[]> it = stream.iterator();

            while (it.hasNext())
            {
                System.out.println("Message : " + new String(it.next().message()));
                consumer.commitOffsets();
            }
        }
    }

    private static ConsumerConfig createConsumerConfig(String a_zookeeper, String a_groupId) {
        Properties props = new Properties();
        props.put("zookeeper.connect", "localhost:2181");
        props.put("group.id", "test-group");
        props.put("zookeeper.session.timeout.ms", "500");
        props.put("zookeeper.sync.time.ms", "200");
        props.put("auto.commit.enable","false");
        props.put("offsets.storage","kafka");
        //props.put("auto.commit.interval.ms", "");

        return new ConsumerConfig(props);
    }

    public static void main(String[] args) {
        String zooKeeper =  "localhost:2181";
        String groupId = "test-group";
        String topic = "test1";
        //int threads = Integer.parseInt(args[3]);

        VulabKafkaConsumer example = new VulabKafkaConsumer(zooKeeper, groupId, topic);
        example.run();
    }
}

Answer 1

我已经处于相同的场景，但使用kafka版本0.9，有必要指定：

auto.offset.reset=earliest

如果此属性未设置，则默认值为＆＃34;最新＆＃34;。此属性用于告诉kafka如果没有初始偏移量或者由于数据已被删除而不再存在当前偏移量时如何操作。我相信你使用的版本需要设置的值是＆＃34;最小的＆＃34;否则默认值为＆＃34;最大＆＃34;

高级消费者手动偏移

1 个答案: