Kafka Producer随机停止+/-不接受修改后的属性

时间:2017-01-06 15:32:23

标签: java windows-10 apache-kafka apache-zookeeper kafka-producer-api

我使用windows 10与kafka v.1.10.1.0一起工作 - 此刻一切都在本地发生,我想说的是我和kafka一起经营一家当地的动物园管理员。我的目标是将一些整数转发给单个经纪人kafka消费者。

这是我当前代码的简化最小示例:

                import kafka.javaapi.producer.Producer;
                import kafka.producer.KeyedMessage;
                import kafka.producer.ProducerConfig;

                import java.io.IOException;
                import java.util.Properties;

                public class simple_example {


                    public static void main(String[] args) throws IOException {

                        long startTime = System.currentTimeMillis();

                        Properties properties = new Properties();

                        //properties.put("retries", 0);
                        properties.put("metadata.broker.list", "localhost:9092");
                        properties.put("serializer.class", "kafka.serializer.StringEncoder");
                        properties.put("request.required.acks", "1");

                        Producer<Integer, String> producer = null;

                        int i = 0;

                        // 10e5 is approx the amount of events i want to forward in the real life version of this
                        while (i < 10e5) {

                            try {

                                producer = new Producer<Integer, String>(new ProducerConfig(properties));

                                    String topic = "test2";
                                    String msg = "Event Number: "+i;

                                    KeyedMessage<Integer, String> data = new KeyedMessage<Integer, String>(topic, msg);
                                    producer.send(data);


                                    System.out.println("-----------------------------------------------------------------------");
                                    long endTime   = System.currentTimeMillis();
                                    long totalTime = endTime - startTime;
                                    System.out.println(totalTime/(1e3)+" [sec] --> ex/s = " + 1/(totalTime/(1e3)));
                                    System.out.println("i= " + i);
                                    System.out.println("-----------------------------------------------------------------------");
                                    i++;
                                    //producer.close();
                                    }  catch (Exception e) {
                                          e.printStackTrace();

                                 } finally {
                                    producer.close();
                                }

                    }
                }

                }

它适用于前几千个事件。但是在8000到10000之间它会停止并发出以下错误:

                            15:23:52,758 INFO  kafka.utils.VerifiableProperties                              - Verifying properties
                            15:23:52,758 INFO  kafka.utils.VerifiableProperties                              - Property metadata.broker.list is overridden to localhost:9092
                            15:23:52,758 INFO  kafka.utils.VerifiableProperties                              - Property request.required.acks is overridden to 1
                            15:23:52,758 WARN  kafka.utils.VerifiableProperties                              - Property retries is not valid
                            15:23:52,758 INFO  kafka.utils.VerifiableProperties                              - Property serializer.class is overridden to kafka.serializer.StringEncoder
                        15:23:52,758 INFO  kafka.client.ClientUtils$                                     - Fetching metadata from broker id:0,host:localhost,port:9092 with correlation id 0 for 1 topic(s) Set(test2)
                        15:23:52,758 INFO  kafka.producer.SyncProducer                                   - Connected to localhost:9092 for producing
                        15:23:52,758 INFO  kafka.producer.SyncProducer                                   - Disconnecting from localhost:9092
                        15:23:52,758 WARN  kafka.client.ClientUtils$                                     - Fetching topic metadata with correlation id 0 for topics [Set(test2)] from broker [id:0,host:localhost,port:9092] failed
                        java.nio.channels.ClosedChannelException
                        at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
                        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
                        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
                        at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
                        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
                        at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
                        at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
                        at kafka.utils.Utils$.swallow(Utils.scala:172)
                        at kafka.utils.Logging$class.swallowError(Logging.scala:106)
                        at kafka.utils.Utils$.swallowError(Utils.scala:45)
                        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
                        at kafka.producer.Producer.send(Producer.scala:77)
                        at kafka.javaapi.producer.Producer.send(Producer.scala:33)
                        at simple_example.main(simple_example.java:40)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:497)
                        at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

我怀疑它可能与我发送的属性有关。多数民众赞成为什么我试图通过额外的那些来修复它,比如为了这样做而被评论的那个我跟着https://kafka.apache.org/documentation.html#brokerconfigs

但是对于所有我改变了(除了代码中的上述三个)我总是得到一个表格的消息:

    16:25:39,810 INFO  kafka.utils.VerifiableProperties - Verifying properties
    16:25:39,810 INFO  kafka.utils.VerifiableProperties - Property metadata.broker.list is overridden to localhost:9092
    16:25:39,810 INFO  kafka.utils.VerifiableProperties - Property request.required.acks is overridden to 1
    16:25:39,810 WARN  kafka.utils.VerifiableProperties - Property retries is not valid

告诉我,我的修改无论如何都无效。

我的问题是

  • 如何让kafka通过所有活动工作?

  • 我应该如何修改kafka的属性以理解和接受它?

非常感谢您提供潜在的帮助,参考,评论

最好,t。

0 个答案:

没有答案