how to get kafka lag using java

Question

I have currently developed a code that would display the topic, partition, and log offset. But I am currently stuck on how to get the Lag of a partition. I know that there is a kafka offset command that does this function but what I need is a java code.

public static void main(String[] args) throws Exception {
    System.out.println("START CONSUMER");final Properties props = new Properties();
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUPID);
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

    // Create the consumer using props.
    final Consumer<Long, String> consumer =  new KafkaConsumer<>(props);

    // Subscribe to the topic.
    int i = 0;
    ArrayList<TopicPartition> partitions = new ArrayList<TopicPartition>();
    for (i=0;i<consumer.partitionsFor(TOPIC).size();i++)
    {
        TopicPartition partitiontemp = new TopicPartition(TOPIC, i);
        partitions.add(partitiontemp);
    }
    consumer.assign(partitions);
    consumer.seekToEnd(partitions);

    for (i=0;i<consumer.partitionsFor(TOPIC).size();i++)
    {
        System.out.printf("Topic: %s partitionID: %d log offset: %d \n", TOPIC, i, consumer.position(partitions.get(i)));
    }

    System.out.printf("CREATE CONSUMER DONE");
    consumer.close();

What I need to do is to output the topic, partition, current offset, log offset, and lag. How can I get the lag for my code or how can I get the current offset for my code. (see image for needed output).

NOTE: I cannot use the (foreach record) functionality because I must not read each record in the input file.

Answer 1

要重现kafka-consumer-groups功能，需要使用方实例和AdminClient实例。

首先，使用AdminClient，您可以调用listConsumerGroupOffsets()来检索主题分区列表和特定组的已提交偏移量。

然后使用使用者获取这些分区的结束偏移量。您使用的方法效率低下，无需分配和查找终点偏移量。您可以简单地致电endOffsets()。

这足以再现屏幕快照中包含的数据。

kafka-consumer-groups还使用AdminClient.describeConsumerGroups()打印分配给每个分区的组成员（如果有）。

Answer 2

您可以通过从消费者那里获取EndOffset来获得LAG

Set<TopicPartition> partitionSet = consumer.assignment();
Map<TopicPartition, Long> endOffsets =consumer.endOffsets(consumer.assignment());

然后遍历设置的地方

for(TopicPartition tp : partitionSet) { LOG.info("Topic :: {} ,EndOffset :: {}, currentOffset {}",tp.topic(),beginningOffsets.get(tp),endOffsets.get(tp), consumer.position(tp)); }

consumer.position（tp）-将获得当前偏移量，将其从endoffset中减去，得到LAG