Question

My Producer使用密钥0,1,2,3,4,5,6,7,8,9

写入10条消息

我的消费者（使用auto commit = false）在第一次循环期间读取消息但是在读取消息时＃34; 2＆＃34;我强迫例外，所以我不承诺。（例外情况仅在第一次循环时抛出 - 存在标志forceError）

在第二次循环期间，消费者应该读取所有消息0,1,2,3,4,5,6,7,8,9和提交但不是这样，为什么？

日志是这样的：

write message with key: 0
write message with key: 1
write message with key: 2
write message with key: 3
write message with key: 4
write message with key: 5
write message with key: 6
write message with key: 7
write message with key: 8
write message with key: 9

1 loop, consumerRecords.count: 10
read message with key: 0
read message with key: 1
read message with key: 2
java.lang.Exception: error don't commit
2 loop, consumerRecords.count(): 0
3 loop, consumerRecords.count(): 0
4 loop, consumerRecords.count(): 0
5 loop, consumerRecords.count(): 0
6 loop, consumerRecords.count(): 0
7 loop, consumerRecords.count(): 0
8 loop, consumerRecords.count(): 0
and so on....

java source：

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Arrays;
import java.util.Properties;
import java.util.UUID;

/*
maven dependency:

<dependencies>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>1.0.0</version>
    </dependency>
</dependencies>

* */
public class KafkaConsumerDemo {
    public static void main(String[] args) {

    String topic = UUID.randomUUID().toString();
    /// producer - write messages with key 0,1,2,3,4,5,6,7,8,9
    {
        Properties properties = new Properties();

        // kafka bootstrap server
        properties.setProperty("bootstrap.servers", "127.0.0.1:9092");
        properties.setProperty("key.serializer", StringSerializer.class.getName());
        properties.setProperty("value.serializer", StringSerializer.class.getName());
        // producer acks
        properties.setProperty("acks", "1");
        properties.setProperty("retries", "3");
        properties.setProperty("linger.ms", "1");

        Producer<String, String> producer = new org.apache.kafka.clients.producer.KafkaProducer<String, String>(properties);


        for (int key = 0; key < 10; key++) {
            System.out.println("write message with key: " + key);
            ProducerRecord<String, String> producerRecord =
                    new ProducerRecord<String, String>(topic, Integer.toString(key), "message that has key: " + Integer.toString(key));
            producer.send(producerRecord);
        }


        producer.close();
    }

      /*consumer -- read message and throw exception during first loop
    first loop read 0,1,2 and force an exception so I don't commit
    second loop should read 0,1,2,3,4,5,6,7,8,9 and commit but is not so :(

     */
    {
        Properties properties = new Properties();

        properties.setProperty("bootstrap.servers", "127.0.0.1:9092");
        properties.setProperty("key.deserializer", StringDeserializer.class.getName());
        properties.setProperty("value.deserializer", StringDeserializer.class.getName());

        properties.setProperty("group.id", "test");
        properties.setProperty("enable.auto.commit", "false");
        properties.setProperty("auto.offset.reset", "earliest");

        KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<String, String>(properties);
        kafkaConsumer.subscribe(Arrays.asList(topic));

        boolean forceError = true;
        int n = 0;
        while (true) {
            n++;
            ConsumerRecords<String, String> consumerRecords = kafkaConsumer.poll(100);
            System.out.println(n + " loop, consumerRecords.count: " + consumerRecords.count());

            try {
                for (ConsumerRecord<String, String> consumerRecord : consumerRecords) {
                    System.out.println("read message with key: " + consumerRecord.key());
                    if (forceError && consumerRecord.key().equals("2")) {
                        forceError = false;
                        throw new Exception("error don't commit");
                    }

                }
                if (!consumerRecords.isEmpty()) {
                    System.out.println("commit ok");
                    kafkaConsumer.commitSync();
                }
            } catch (Exception e) {
                System.out.println(e);
            }

        }
    }
}
}

Answer 1

输出中的关键行是：

1 loop, consumerRecords.count: 10

基本上，对poll()的第一次调用返回了所有10条记录。消费者已收到所有消息并在ConsumerRecord容器中将消息传递给您。然后由你来做你想做的事，但从消费者的角度来看，它已经完成了它的工作。

即使您没有提交，消费者也会跟踪其当前位置。它的当前位置是偏移11，它是日志的结尾，所以在为主题生成新消息之前它不会接收任何内容。

由于您没有提交并将重置策略设置为earliest，如果您重新启动消费者（或取消订阅并重新订阅），它将再次收到所有10条消息。

Apache Kafka - Autocommit = false，没有提交

1 个答案: