处理Kafka CommitFailedException

Question

我正在使用kafka-clients 2.0.0版。

我在单个消费者组下有3个kafka消费者，具有默认的消费者参数和一个带有3个分区的主题。

消费者每秒轮询一次数据，消耗50条消息的时间不到1.5秒。

进行重新平衡时，我得到：

org.apache.kafka.clients.consumer.CommitFailedException: 
Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member.This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. 
You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

我尝试将max.poll.interval.ms增加到Integer.MAX_VALUE，但在重新平衡时仍然遇到此异常。

我还尝试实现RebalanceListener和Interceptor来跟踪轮询已完成，但是在重新平衡之前尚未提交。但这也无济于事。

以下是一些代码可以测试此行为：

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

import java.net.InetAddress;
import java.net.UnknownHostException;
import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

public class Conzumer {

    private KafkaConsumer consumer;


    public static void main(String[] args) {
        ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
        Conzumer task = new Conzumer();
        task.init();
        scheduler.scheduleAtFixedRate(() -> task.doWork(), 5, 1, TimeUnit.SECONDS);
    }

    public void doWork() {
        try {
//            System.out.println(LocalDateTime.now() + " Poll triggered");
            ConsumerRecords<String, String> consumerRecords = consumer.poll(Duration.ofSeconds(0));
            if (!consumerRecords.isEmpty()) {
//                System.out.println(LocalDateTime.now() + " polled " + consumerRecords.count() + " in " + pollElapsed + " ms");
            } else {
//                System.out.println(LocalDateTime.now() + " Zero batch polled");
            }
            consumer.commitSync();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public void init() {

        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "");
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringDeserializer.class);
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringDeserializer.class);
        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, String.valueOf(Boolean.FALSE));
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        properties.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "5000");
        properties.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, "8192");
        try {
            properties.put(ConsumerConfig.CLIENT_ID_CONFIG, InetAddress.getLocalHost().getHostName());
        } catch (UnknownHostException e) {}

        properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 50);
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "local");
        this.consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Arrays.asList("topicName"));
    }


}

只需运行几次主类（3个就足够了），然后关闭几个应用程序。

有人可以告诉我此异常的根本原因还有如何避免它。

还可以确保，如果我忽略此异常，则不会丢失我的消息，并且在重新平衡后，它们将在下一个民意调查中返回一次吗？