Kafka在启动时不尊重主题的保留期限?

时间:2016-09-15 14:12:34

标签: apache-kafka apache-zookeeper

我们在测试计算机上运行一个ZooKeeper服务器和一个Kafka代理。我们使用的是ZooKeeper 3.3.5和Kafka 0.9.0.1。

代理上的邮件保留期限设置为4小时:

log.retention.ms = null
log.retention.bytes = -1
log.retention.hours = 4
log.retention.minutes = null
log.retention.check.interval.ms = 30000

我们有一个主题,其保留期基本上是无限的(我们将其设置为50年)。

在某些时候我们关闭然后重新启动ZooKeeper,然后我们关闭并重新启动Kafka经纪人。

查看日志,看起来Kafka经纪人在启动时决定删除旧邮件,即使它们在50年的保留期内也是如此。

首先,这是Kafka正常启动:

[2016-09-15 09:09:24,877] INFO starting (kafka.server.KafkaServer)
[2016-09-15 09:09:24,883] INFO Connecting to zookeeper on 10.0.4.83:2182 (kafka.server.KafkaServer)
[2016-09-15 09:09:25,056] INFO Loading logs. (kafka.log.LogManager)
[2016-09-15 09:09:25,096] INFO Completed load of log ecosystem_sharedmodel-0 with log end offset 102152 (kafka.log.Log)
[2016-09-15 09:09:25,105] INFO Logs loading complete. (kafka.log.LogManager)
[2016-09-15 09:09:25,106] INFO Starting log cleanup with a period of 30000 ms. (kafka.log.LogManager)
[2016-09-15 09:09:25,107] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-09-15 09:09:25,149] INFO Awaiting socket connections on ip-10-0-4-83:9092. (kafka.network.Acceptor)
[2016-09-15 09:09:25,151] INFO [Socket Server on Broker 1], Started 1 acceptor threads (kafka.network.SocketServer)
[2016-09-15 09:09:25,169] INFO [ExpirationReaper-1], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-09-15 09:09:25,173] INFO [ExpirationReaper-1], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-09-15 09:09:25,244] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-09-15 09:09:25,250] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-09-15 09:09:25,250] INFO 1 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-09-15 09:09:25,321] INFO [GroupCoordinator 1]: Starting up. (kafka.coordinator.GroupCoordinator)

然后,30秒后,日志清理开始并删除我们的所有消息:

[2016-09-15 09:09:55,114] INFO Rolled new log segment for 'ecosystem_sharedmodel-0' in 3 ms. (kafka.log.Log)
[2016-09-15 09:09:55,115] INFO Scheduling log segment 0 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,115] INFO Scheduling log segment 618 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,115] INFO Scheduling log segment 3052 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 6050 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 73419 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 73553 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 82663 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 85600 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 96316 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 99030 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 101756 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:10:55,116] INFO Deleting segment 618 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,116] INFO Deleting segment 0 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,116] INFO Deleting segment 6050 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting segment 73419 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting segment 3052 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000073419.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,117] INFO Deleting segment 73553 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000003052.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,118] INFO Deleting segment 82663 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,118] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000073553.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,118] INFO Deleting segment 85600 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,118] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000082663.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,119] INFO Deleting segment 96316 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,119] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000085600.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,119] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000096316.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,119] INFO Deleting segment 99030 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,119] INFO Deleting segment 101756 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,120] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000000000.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,121] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000099030.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,121] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000000618.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,121] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000101756.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,123] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000006050.index.deleted (kafka.log.OffsetIndex)

这是卡夫卡的预期行为吗?我们这边有误会吗?配置问题?

编辑:结果是用户错误,卡夫卡方面没有错误。

0 个答案:

没有答案