Kafka经纪人问题(最大大小的副本管理器)

时间:2020-02-25 23:08:12

标签: apache-kafka

我的kafka环境中出现以下错误。它可以工作几个小时,然后窒息。

20200224;21:01:38: [2020-02-24 21:01:38,615] ERROR [ReplicaManager broker=0] Error processing fetch with max size 1048576 from consumer on partition SANDBOX.BROKER.NEWORDER-0: (fetchOffset=211886, logStartOffset=-1, maxBytes=1048576, currentLeaderEpoch=Optional.empty) (kafka.server.ReplicaManager)

20200224;21:01:38: org.apache.kafka.common.errors.CorruptRecordException: Found record size 0 smaller than minimum record overhead (14) in file /data/tmp/kafka-topic-logs/SANDBOX.BROKER.NEWORDER-0/00000000000000000000.log.

20200224;21:05:48: [2020-02-24 21:05:48,711] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

20200224;21:10:22: [2020-02-24 21:10:22,204] INFO [GroupCoordinator 0]: Member xxxxxxxx_011-9e61d2c9-ce5a-4231-bda1-f04e6c260dc0-StreamThread-1-consumer-27768816-ee87-498f-8896-191912282d4f in group yyyyyyyyy_011 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)

设置: 1. Kafka经纪人(kafka_2.12-2.1.1 /) 1. Zookeeper

配置kafka:

num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

log.dirs=/data/tmp/kafka-topic-logs
num.partitions=1
num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.flush.interval.ms=1000

log.retention.hours=168
log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=6000


group.initial.rebalance.delay.ms=0
delete.topic.enable=true

zookeper配置

    dataDir=/data/tmp/zookeeper
    clientPort=2181
    maxClientCnxns=0
    delete.topic.enable=true

0 个答案:

没有答案