Question

我正在尝试使用卡夫卡。我已经开始

ScrollView

我通过kafka-producer发送消息，并在kafka-console-consumer上成功接收。现在，我想一次产生和消费大约5000条消息。我研究了文档，并知道有两个命令。

kafka-console-producer and 
kafka-console-consumer.

我尝试使用这些命令。

kafka-verifiable-producer.sh

kafka-verifiable-consumer.sh

结果如下

 kafka-verifiable-producer.sh --broker-list localhost:9092 --max-messages 5000 --topic data-sending

kafka-verifiable-consumer.sh  --group-instance-id 1 --group-id data-world --topic data-sending --broker-list localhost:9092

在使用者控制台上，结果如下

"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4996","offset":44630,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4997","offset":44631,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4998","offset":44632,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4999","offset":44633,"topic":"try_1","partition":0}

{"timestamp":1581268289769,"name":"shutdown_complete"}
 {"timestamp":1581268289771,"name":"tool_data","sent":5000,"acked":5000,"target_throughput":-1,"avg_throughput":5285.412262156448}

在以上结果中，键为null。我如何使用此命令发送大量消息？我尝试研究一个如何使用它们的示例，但没有找到任何示例。它产生像值一样的整数，但是我可以在哪里插入消息？有什么方法可以使用此命令批量生成消息？是否有可能在Windows中实现此类命令，或者仅适用于Linux？到示例的任何链接将不胜感激。

Answer 1

脚本kafka-verifiable-producer.sh执行类org.apache.kafka.tools.VerifiableProducer。（https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java）

其程序参数--throughput，--repeating-keys和--value-prefix可以满足您的需求。

例如，以下代码产生带有前缀值111的消息，并带有增量密钥，该密钥每5条消息旋转一次。您也可以使用--throughput选项配置消息的吞吐量。在此示例中，它平均每秒产生5条消息。

./kafka-verifiable-producer.sh --broker-list localhost:9092 --max-messages 10 --repeating-keys 5 --value-prefix 100 --throughput 5 --topic test
{"timestamp":1581271492652,"name":"startup_complete"}
{"timestamp":1581271492860,"name":"producer_send_success","key":"0","value":"100.0","offset":45,"topic":"test","partition":0}
{"timestamp":1581271492862,"name":"producer_send_success","key":"1","value":"100.1","offset":46,"topic":"test","partition":0}
{"timestamp":1581271493048,"name":"producer_send_success","key":"2","value":"100.2","offset":47,"topic":"test","partition":0}
{"timestamp":1581271493254,"name":"producer_send_success","key":"3","value":"100.3","offset":48,"topic":"test","partition":0}
{"timestamp":1581271493256,"name":"producer_send_success","key":"4","value":"100.4","offset":49,"topic":"test","partition":0}
{"timestamp":1581271493457,"name":"producer_send_success","key":"0","value":"100.5","offset":50,"topic":"test","partition":0}
{"timestamp":1581271493659,"name":"producer_send_success","key":"1","value":"100.6","offset":51,"topic":"test","partition":0}
{"timestamp":1581271493860,"name":"producer_send_success","key":"2","value":"100.7","offset":52,"topic":"test","partition":0}
{"timestamp":1581271494063,"name":"producer_send_success","key":"3","value":"100.8","offset":53,"topic":"test","partition":0}
{"timestamp":1581271494268,"name":"producer_send_success","key":"4","value":"100.9","offset":54,"topic":"test","partition":0}
{"timestamp":1581271494483,"name":"shutdown_complete"}
{"timestamp":1581271494484,"name":"tool_data","sent":10,"acked":10,"target_throughput":5,"avg_throughput":5.452562704471101}

最简单的方法是修改/扩展上述类，如果您正在寻找更多自定义的消息键和值。

卡夫卡可验证的生产者和消费者问题

1 个答案: