卡夫卡减少消费者的滞后

Question

我正在设置新的Kafka集群，出于测试目的，我创建了具有1个分区和3个副本的主题。

现在，我通过生产者并行发送消息，例如每秒发送5万条消息。而且我在一个组内创建了一个“使用者”，它只能每秒提取3万条消息。

我可以更改主题级别，分区级别，使用者级别的配置。

我正在通过grafana + prometheus监视所有内容。

任何想法，哪种配置或其他东西可以帮助我消费更多数据？

预先感谢

Answer 1

A Kafka consumer polls the broker for messages and fetches whatever messages are available for consumption, depending upon the consumer configuration used. In general, it is efficient to transfer as much data is possible in a single poll request if increasing throughput is your aim. But how much data is transferred in a single poll is determined by the size of messages, number of records and some parameters which control how much time to wait for messages to be available.

In general, you can influence throughput using one or more of the following consumer configurations:

fetch.min.bytes

max.partition.fetch.bytes

fetch.max.bytes

max.poll.records

fetch.max.wait.ms