Question

我最近开始使用kafka来阅读通过网络抓取工具发送的文档。我注意到的是，当我处理几百万份文件时，消费者一遍又一遍地处理相同的消息。看起来数据由于某种原因没有得到承诺。当我用几百条消息测试消费者时，情况并非如此。

我在java中使用kafka高级消费者客户端代码。我使用的消费者组运行的线程数相当于分区数。因此每个线程都被分解为一个分区。这是一个用于轮询数据的代码段。

while (true) { try{ if(consumerDao.canPollTopic()){ ConsumerRecords records = consumer.poll(this.config.getPropertyAsIneger(IPreProcessorConstant.KAFKA_POLL_COUNT)); for (ConsumerRecord record : records) { if(record.value()!=null){ TextAnalysisRequest textAnalysisObj = record.value(); if(textAnalysisObj!=null){ PostProcessRequest req = new PostProcessRequest(); req.setRequest(this.getRequest(textAnalysisObj)); PreProcessorUtil.submitPostProcessRequest(req, config); } } } }else{ Thread.sleep(this.config.getPropertyAsIneger(IPreProcessorConstant.KAFKA_POLL_SLEEP)); } }catch(Exception ex){ LOGGER.error("Error in Full Consumer group worker", ex); } }


consumer.auto.commit=true
consumer.auto.commit.interval=1000
consumer.session.timeout=180000
consumer.poll.records=2147483647
consumer.request.timeout=181000

这是我设置的kafka消费者配置参数。休息是默认值。


metric.reporters = 
metadata.max.age.ms = 300000
partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
max.partition.fetch.bytes = 1048576
bootstrap.servers = [kafkahost1:9092, kafkahost2:9092]
ssl.keystore.type = JKS
enable.auto.commit = true
sasl.mechanism = GSSAPI
interceptor.classes = null
exclude.internal.topics = true
ssl.truststore.password = null
client.id =ssl.endpoint.identification.algorithm = null
max.poll.records = 2147483647
check.crcs = true
request.timeout.ms = 181000
heartbeat.interval.ms = 3000
auto.commit.interval.ms = 1000
receive.buffer.bytes = 65536
ssl.truststore.type = JKS
ssl.truststore.location = null
ssl.keystore.password = null
fetch.min.bytes = 1
send.buffer.bytes = 131072
value.deserializer = class com.test.preprocessor.consumer.serializer.KryoObjectSerializer
group.id = full_group
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
ssl.key.password = null
fetch.max.wait.ms = 500
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
session.timeout.ms = 180000
metrics.num.samples = 2
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
auto.offset.reset = latest

这里是完整的消费者配置：

我的样本kafka队列有8个分区，有2个复制因子。

server.properties中的日志保留期设置为168小时。 log.retention.hours=168 log.roll.hours=168

{{1}}

不知道我在这里失踪了什么。

Answer 1

我猜想分区分配有问题。

Answer 2

我将我的消费者资源中的auto.commit.interval.ms=8000从3000增加到8000。这修复了重复记录问题。

Kafka消费者多次接收相同的消息

2 个答案: