Question

我要求证明kafka生产者可以生产100万消息/秒到Kafka集群，然后评估其性能。

如何实现100万条消息/秒？

j'ai une选择了commeçapourune producer kafka：

public static void main(String args[]){

    Random rnd = new Random();

    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    props.put("acks", "all");
    props.put("retries", 0);
    props.put("batch.size", 16384);
    props.put("linger.ms", 1);
    props.put("buffer.memory", 33554432);
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    Producer<String, String> producer = new KafkaProducer<String, String>(props);
    int counter=0;
    int i = 0;
    while ( true ){
        TimeZone tz = TimeZone.getTimeZone("UTC");
        DateFormat df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm.sss'Z'"); // Quoted "Z" to indicate UTC, no timezone offset
        df.setTimeZone(tz);
        String nowAsISO = df.format(new Date());
        ++counter;

            final String message = "sensor" + i + ":" + Integer.toString(rnd.nextInt(10000)) + " " +  String.valueOf(rnd.nextDouble())+ " " +
                    "MyDevice" + " " +
                    "Sensor" + " " + "Sensing" + " " + "Property" + " " + "Unit" + " " + "9845A" + " " + nowAsISO ;


        try {
            producer.send(new ProducerRecord<String, String>("test", message), new Callback() {
                        public void onCompletion(RecordMetadata recordMetadata, Exception e) {
                            if ( e == null ){
                                System.out.println("Partition: "+recordMetadata.partition()
                                        +", Offset" + recordMetadata.offset()
                                        + ", timestamp: " + recordMetadata.timestamp());
                                System.out.println(message);
                            }
                            else {
                                e.printStackTrace();
                            }
                        }
                    }
            );

            i++;
            TimeUnit.SECONDS.sleep(1000);
        }
        catch ( InterruptedException e   ){
            System.out.println("I was interrupted.");
        }
    }

}

}

谢谢！

Answer 1

有一些说明可以在librdkafka github页面上只有2个代理生成850,000 msg /秒

https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md

如果您有足够的网络带宽和磁盘I / O容量，您应该能够通过3个代理获得超过100万msg /秒。

使用kafka生成100万条消息/秒

1 个答案: