Question

我正在尝试获取每个可用分区的当前偏移量。根据文档，consumer.position应该可以解决问题，所以我尝试这样做：

@RawQuery
abstract int simpleRawQuery(SupportSQLiteQuery sqliteQuery)

@Transaction
public int deleteData(List<Long> pkList) {
   SimpleSQLiteQuery query = new SimpleSQLiteQuery("DELETE FROM tb WHERE _id IN (" + StringUtils.join(pkList,",") + ")";
   return simpleRawQuery(query) 
}

但是，consumer = Consumer({ 'bootstrap.servers': config.BOOTSTRAP_SERVERS, 'group.id': config.CONSUMER_GROUP, 'enable.auto.commit': False, }) # get all topics topics = consumer.list_topics() # get all partitions partitions = [] for name, meta in topics.topics.items(): for partition_id in meta.partitions.keys(): part = TopicPartition(name, partition_id) partitions.append(part) # get all offsets x = consumer.position(partitions)中结果分区中的所有偏移量仍为x。

如果我使用镜头或其他工具进行检查，则可以看到此结果不正确，这是我取消演唱的消费群体已消费了消息并将其提交给Kafka。

Answer 1

尝试添加consumer.subscribe()或consumer.assign()函数

consumer = Consumer({
    'bootstrap.servers': config.BOOTSTRAP_SERVERS,
    'group.id': config.CONSUMER_GROUP,
    'enable.auto.commit': False,
})

# get all partitions
partitions = []
for name, meta  in topics.topics.items():
    for partition_id in meta.partitions.keys():
        part = TopicPartition(name, partition_id)
        partitions.append(part)

consumer.assign(partitions)
committed = consumer.committed(tp)

last_offset = consumer.position(tp)
print("topic: %s partition: %s committed: %s last: %s lag: %s" % (TOPIC, p, committed, last_offset))

Answer 2

作为参考，这是有效的解决方案：

consumer = Consumer({
    'bootstrap.servers': config.BOOTSTRAP_SERVERS,
    'group.id': config.CONSUMER_GROUP,
    'enable.auto.commit': False,
})

# get all topics
topics = consumer.list_topics()

# get all partitions
partitions = []
for name, meta  in topics.topics.items():
    for partition_id in meta.partitions.keys():
        part = TopicPartition(name, partition_id)
        partitions.append(part)

# get last committed offsets
partitions = consumer.committed(partitions)

显然，consumer.position不能像宣传的那样工作，但是consumer.committed会返回存储的偏移量，即使消费者当前未订阅主题/分区也是如此。

如何为我的消费者获取所有分区的当前偏移量？

2 个答案: