Question

我正在使用kafka_2.12-1.1.0。以下是我的保留权限。我正在使用Win OS。

log.retention.ms=120000

log.segment.bytes=2800

log.retention.check.interval.ms=30000

我看到日志被翻转并生成* 0000.log到* 0069.log，然后生成* 00103.log等，然后我希望它删除旧文件，但是当保留逻辑启动时，我得到以下错误过程无法访问该文件，因为该文件正在被另一个进程使用，并且kafka服务器崩溃并且无法从那里恢复。尝试删除00000000000000000000.log时会发生这种情况？您能告诉我一些命令来检查它们是否为活动段，或者为什么要删除该段并使其崩溃。

PFB日志文件（如果要查看）。

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 39

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 39 in 39 ms. 

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 69 

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 69 in 52 ms.

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 91

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 91 in 18 ms. 

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Found deletable segments with base offsets [0,69] due to retention time 120000ms breach (kafka.log.Log)

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 103 (kafka.log.ProducerStateManager)

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 103 in 25 ms. (kafka.log.Log)

 [Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Scheduling log segment [baseOffset 0, size 2746] for deletion. (kafka.log.Log)

ERROR Error while deleting segments for myLocalTopic-0 in dir D:\tmp\kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log -> D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.
.............   at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:98)
.......................................................
 ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for myLocalTopic-0 in dir D:\tmp\kafka-logs
Caused by: java.nio.file.FileSystemException: D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log -> D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

Click here for file list image

Answer 1

Apache Kafka为我们提供以下保留政策

基于时间的保留
基于大小的保留

要基于时间的保留，请尝试以下命令，

设置保留时间的命令

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.ms=1680000

删除主题级别保留时间的命令

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.ms

对于基于大小的保留，请尝试以下命令，设置保留大小：

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.bytes=104857600

要删除，

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.bytes

Apache Kafka日志保留配置问题或如何配置Kafka保留策略？

1 个答案: