如何为Springframework Kafka使用者启用Kafka补偿？

Question

当服务器无法通过消费者访问权限连接到kafka主题时，我看到大量错误消息和警告消息被写入日志。 我每秒看到数百条日志。

这里的根本问题是权限不正确：目前我没有正在使用的CN的消费者访问权限。我正在解决此问题，但是我想确保，如果出现此类连接问题，我再也不会出现千兆字节的日志了。

我曾尝试按照建议将ConsumerConfig.RECONNECT_BACKOFF_MS_CONFIG和ConsumerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG添加到Consumer Factory的属性中，但是配置似乎没有任何作用。

在一天结束时，我希望减少连接尝试的次数，直到您每5分钟仅看到一次。

重复的错误是：

17:38:33.068 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] ERROR o.s.k.listener.LoggingErrorHandler -                                                                                          
                Error while processing: null                                                                                                                                                                        
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [payments.transaction.submit]                                                                                          
17:38:33.069 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN  o.a.k.c.consumer.internals.Fetcher -                                                                                          
                [Consumer clientId=consumer-2, groupId=xxxxx] Not authorized to read from topic payments.transaction.submit.                                                                              
17:38:33.069 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] ERROR o.s.k.listener.LoggingErrorHandler -                                                                                          
                Error while processing: null                                                                                                                                                                        
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [payments.transaction.submit]                                                                                          
17:38:33.069 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN  o.a.k.c.consumer.internals.Fetcher -                                                                                          
                [Consumer clientId=consumer-2, groupId=xxxxx] Not authorized to read from topic payments.transaction.submit.                                                                              
17:38:33.069 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] ERROR o.s.k.listener.LoggingErrorHandler -                                                                                          
                Error while processing: null                                                                                                                                                                        
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [payments.transaction.submit]                                                                                          
17:38:33.070 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN  o.a.k.c.consumer.internals.Fetcher -                                                                                          
                [Consumer clientId=consumer-2, groupId=xxxxx] Not authorized to read from topic payments.transaction.submit.                                                                              
17:38:33.070 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] ERROR o.s.k.listener.LoggingErrorHandler -                                                                                          
                Error while processing: null

消费者：

@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    ~~~~~~~

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafakBrokerUrl + ":" + kafkaBrokerPort);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG, 1000L); // 1 Sec
        props.put(ConsumerConfig.RECONNECT_BACKOFF_MS_CONFIG, 10 * 1000L); // 10 Sec
        props.put(ConsumerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, 300 * 1000L); // 5 Min

        props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, kafkaTruststorePath);
        props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, kafkaTruststorePW);
        props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, kafkaKeystorePath);
        props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, kafkaKeystorePW);
        props.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, kafkaSslKeyPW);
        props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");

        return new DefaultKafkaConsumerFactory<>(props);
    }

    @Bean
    public KafkaListenerContainerFactory<?> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}

Kafka监听器：

@Component
public class Listener {
    private final Logger audit = LoggerFactory.getLogger("auditLogger");

    @KafkaListener(topics = "bob.the.topic", groupId = "ConsumerGroupBob")
    public void listen(@Payload String message) throws Exception {
        ***** DO STUFF HERE ****
    }
}

POM.xml

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.7.RELEASE</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>
...