Question

我有两个kafka群集。我需要使用kafka-spring在它们之间实现某种同步。

[cluster A, topic A]  <-- [spring app] --> [cluster B, topic B]

我创建了标注为@Transactional的侦听器，该侦听器使用kafkaTemplate发布消息。当同时连接到两个群集时，此方法非常适用。与目标群集的连接丢失时-侦听器似乎仍在确认新消息，但未发布。我尝试了对侦听器进行手动确认，禁用自动提交等功能，但是似乎它们无法正常运行。.当连接重新联机时，消息将永远无法传递。需要帮助。

    @KafkaListener(topics = "A", containerFactory = "syncLocalListenerFactory")
    public void consumeLocal(@Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) String key, @Payload SyncEvent message, Acknowledgment ack) {
        kafkaSyncRemoteTemplate.send("B", key, message);
        ack.acknowledge();
    }

我正在获取日志：

2019-04-26 12:11:40.808  WARN 21304 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1001 could not be established. Broker may not be available.
2019-04-26 12:11:40.828  WARN 21304 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-2, groupId=app-sync] Connection to node 1001 could not be established. Broker may not be available.
2019-04-26 12:11:47.829 ERROR 21304 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='...' and payload='...' to topic B:

org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for sync-2: 30002 ms has passed since batch creation plus linger time

2019-04-26 12:11:47.829 ERROR 21304 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='...' and payload='...' to topic B:

org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for sync-2: 30002 ms has passed since batch creation plus linger time

---编辑---

kafkaProperties是从application.properties文件读取的默认kafka-spring属性，但在这种情况下，它们都是默认值

    @Bean
    public ConsumerFactory<String, SyncEvent> syncLocalConsumerFactory() {
        Map<String, Object> config = kafkaProperties.buildConsumerProperties();

        config.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaProperties.getStreams().getApplicationId() + "-sync");
        config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
        config.put(JsonDeserializer.VALUE_DEFAULT_TYPE, SyncEvent.class);
        config.put(JsonDeserializer.TRUSTED_PACKAGES, "app.structures");

        config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

        DefaultKafkaConsumerFactory<String, SyncEvent> cf = new DefaultKafkaConsumerFactory<>(config);
        cf.setValueDeserializer(new JsonDeserializer<>(SyncEvent.class, objectMapper));
        return cf;
    }

    @Bean(name = "syncLocalListenerFactory")
    public ConcurrentKafkaListenerContainerFactory<String, SyncEvent> kafkaSyncLocalListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, SyncEvent> factory = new ConcurrentKafkaListenerContainerFactory();
        factory.setConsumerFactory(syncLocalConsumerFactory());
        factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
        factory.getContainerProperties().setAckOnError(false);
        factory.setErrorHandler(new SeekToCurrentErrorHandler(0));
        return factory;
    }

Answer 1

This website介绍了如何设置错误处理程序（使用SeekToCurrentErrorHandler），它可能会对您有所帮助。来自Spring documentation：

SeekToCurrentErrorHandler：一个错误处理程序，它寻求以下位置中每个主题的当前偏移量其余记录。用于在消息后回滚分区失败，以便可以重播。

Answer 2

之所以会发生这种情况，是因为kafka事务无法跨越集群。您的@Transactional注释没有任何意义，因此偏移量将提交给群集A，无论是否成功发布到群集B。

您当前可以为跨集群流获得的最佳保证是“ Atleast-once”处理，并且可以通过确保偏移量仅在集群B的目标代理确认消息后才将其提交给集群A来实现。

有关更多信息，请参阅我的博客文章-https://medium.com/@harelopler/kafka-cross-cluster-stream-reaching-at-least-once-semantics-c74ed0eb1a54

春天，kafka群集之间的事务转移消息

2 个答案: