Question

我正在寻找一种方法来使用具有特定偏移范围的Kafka主题消耗某些消息（假设我的分区的偏移量为200-300，我想使用250-270的偏移量来消耗消息）

我正在使用下面的代码来指定初始偏移量，但是它将占用从250到结束的所有消息。是否有任何方法/属性可用于设置结束偏移量以消耗直到该点的消息。

    @KafkaListener(id = "KafkaListener",
        topics = "${kafka.topic.name}", 
        containerFactory = "kafkaManualAckListenerContainerFactory", 
        errorHandler = "${kafka.error.handler}",
        topicPartitions = @TopicPartition(topic = "${kafka.topic.name}",
        partitionOffsets = {
                @PartitionOffset(partition = "0", initialOffset = "250"), 
                @PartitionOffset(partition = "1", initialOffset = "250")
        }))

Answer 1

KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<String, String>(properties);

boolean keepOnReading = true;

// offset to read the data from.
long offsetToReadFrom = 250L; 

// seek is mostly used to replay data or fetch a specific message
// seek
kafkaConsumer.seek(partitionToReadFrom, offsetToReadFrom);

while(keepOnReading) {
   ConsumerRecords<String, String> records = kafkaConsumer.poll(Duration.ofMillis(100));

   for (ConsumerRecord<String, String> record : records) {
      numberOfMessagesRead ++;
      logger.info("Key: "+record.key() + ", Value: " + record.value());
      logger.info("Partition: " + record.partition() + ", Offset: " + record.offset());

      if(record.offset() == 270L) {
        keepOnReading = false;
        break;
      }
   }
}

我希望这对您有帮助！

Answer 2

您可以使用seek()强制消费者开始从特定的偏移量开始消费，然后从poll()开始消费，直到达到目标最终偏移量为止。

public void seek(TopicPartition partition, long offset)

覆盖使用者将在下一个poll(timeout)上使用的提取偏移量。如果为此API调用了同一分区不止一次，最新的偏移量将用于下一个poll()。请注意，如果任意使用此API，则可能会丢失数据在消费过程中使用，以重置获取偏移量

例如，假设您要从偏移量200开始：

TopicPartition tp = new TopicPartition("myTopic", 0);
Long startOffset = 200L
Long endOffset = 300L

List<TopicPartition> topics = Arrays.asList(tp);
consumer.assign(topics);
consumer.seek(topicPartition, startOffset);

现在，您只需要保持poll()到到达endOffset：

boolean run = true;
while (run) {
    ConsumerRecords<String, String> records = consumer.poll(1000);
    for (ConsumerRecord<String, String> record : records) {

        // Do whatever you want to do with `record`

        // Check if end offset has been reached
        if (record.offset() == endOffset) {
            run = false;
            break;
        }
    }
}

阅读来自Kafka主题的一系列偏移量之间的消息

2 个答案: