Question

我是kafka的新手，我尝试在kafka中使用COMPACT模式。我看看kafka文档，我认为为COMPACT调制解调器设置log.cleaner.enable=true和log.cleanup.policy=compact，但它似乎不起作用。请你帮助我好吗。感谢。

环境

Ubuntu 14.04.3 LTS

Kafka kafka_2.11-0.8.2.2

kafka config

log.retention.ms = 10000
log.retention.check.interval.ms = 1000
log.cleaner.enable =真
log.cleaner.delete.retention.ms = 1000
log.cleaner.backoff.ms = 1000
log.cleaner.min.cleanable.ratio = 0.01
log.cleanup.policy =紧凑

在kafka日志中，我看到'INFO [kafka-log-cleaner-thread-0]，Starting（kafka.log.LogCleaner）'，所以我认为成功启用COMPACT模式。

我按照快速启动kafka文档，并输入一些数据来测试主题。

root@50d8fe84c573:~/kafka_2.11-0.8.2.2# date
Sun Jan 10 14:23:26 UTC 2016

root@50d8fe84c573:~/kafka_2.11-0.8.2.2# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
[2016-01-10 14:23:38,312] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
22:23:45
22:24:05
22:25:00

root@50d8fe84c573:~/kafka_2.11-0.8.2.2# date
Sun Jan 10 14:25:21 UTC 2016
root@50d8fe84c573:~/kafka_2.11-0.8.2.2# ./bin/kafka-console-consumer.sh --zookeeper localhost --topic test --from-beginning --property print.key=true
null 22:23:45
null 22:24:05
null 22:25:00

root@50d8fe84c573:~/kafka_2.11-0.8.2.2# date
Sun Jan 10 14:30:06 UTC 2016
root@50d8fe84c573:~/kafka_2.11-0.8.2.2# ./bin/kafka-console-consumer.sh --zookeeper localhost --topic test --from-beginning --property print.key=true
null 22:23:45
null 22:24:05
null 22:25:00

我希望只保留相同的密钥，因此将删除22:23:45和22:24:05的消息。

感谢摩根的建议。我一直在推动有关键的消息。

root@9d8bdae48429:~/kafka_2.11-0.8.2.2# ./bin/kafka-topics.sh --zookeeper localhost --topic test --delete
root@9d8bdae48429:~/kafka_2.11-0.8.2.2# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

root@9d8bdae48429:~/kafka_2.11-0.8.2.2# date
Wed Jan 13 01:21:59 UTC 2016
root@9d8bdae48429:~/kafka_2.11-0.8.2.2# bpython3
from kafka import (
    KafkaClient, KeyedProducer,
    Murmur2Partitioner, RoundRobinPartitioner)
kafka = KafkaClient('localhost:9092')
producer = KeyedProducer(kafka)

>>> producer.send_messages(b'test', b'key', b'09:22:08')
[ProduceResponse(topic=b'test', partition=0, error=0, offset=0)]
>>> producer.send_messages(b'test', b'key', b'09:22:28')
[ProduceResponse(topic=b'test', partition=0, error=0, offset=1)]
>>> producer.send_messages(b'test', b'key', b'09:23:00')
[ProduceResponse(topic=b'test', partition=0, error=0, offset=2)]

root@9d8bdae48429:~/kafka_2.11-0.8.2.2# date
Wed Jan 13 01:23:05 UTC 2016
root@9d8bdae48429:~/kafka_2.11-0.8.2.2# ./bin/kafka-console-consumer.sh --zookeeper localhost --topic test --from-beginning --property print.key=true
key 09:22:08
key 09:22:28
key 09:23:00
^CConsumed 3 messages
root@9d8bdae48429:~/kafka_2.11-0.8.2.2# date
Wed Jan 13 01:27:18 UTC 2016
root@9d8bdae48429:~/kafka_2.11-0.8.2.2# ./bin/kafka-console-consumer.sh --zookeeper localhost --topic test --from-beginning --property print.key=true
key 09:22:08
key 09:22:28
key 09:23:00

但结果是，测试主题仍保留所有消息。

我不知道我错过了什么。请你帮助我好吗？谢谢。： - ）

Kafka 0.8.2.2 - 紧凑模式不起作用

环境

kafka config

0 个答案: