Question

build.sbt的一部分

scalaVersion := "2.11.0"

val sparkVersion = "2.1.1";

libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided";
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided";

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.2";
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-8" % sparkVersion;

制片人代码：

        updates.foreachPartition{ partition =>
            val kafkaProperties = new Properties();
            kafkaProperties.put("serializer.class","kafka.serializer.StringEncoder");
            kafkaProperties.put("metadata.broker.list",kafkaBrokers);
            //kafkaProperties.put("partitioner.class","RuleUpdateKafkaPartitioner");
            val kafkaConfig = new ProducerConfig(kafkaProperties);
            val kafkaProducer = new Producer[String,String](kafkaConfig);
            partition.foreach { record =>
                ...
                val data = new KeyedMessage[String,String](kafkaTopic,record.ruleID(),record_json.toString());
                kafkaProducer.send(data);
            }
            kafkaProducer.close();

错误：

17/06/16 17:52:05 INFO VerifiableProperties：验证属性 17/06/16 17:52:05 INFO VerifiableProperties：Property metadata.broker.list被重写为10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties：属性serializer.class是重写为kafka.serializer.StringEncoder 17/06/16 17:52:05 WARN AppInfo $：无法从MANIFEST.MF读取Kafka版本。可能的原因：java.lang.NullPointerException 17/06/16 17:52:05 INFO VerifiableProperties：验证属性17/06/16 17:52:05 INFO VerifiableProperties：属性metadata.broker.list是被覆盖到10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties：属性serializer.class被重写为 kafka.serializer.StringEncoder 17/06/16 17:52:05 INFO VerifiableProperties：验证属性17/06/16 17:52:05 INFO VerifiableProperties：属性metadata.broker.list被重写为 10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties：属性serializer.class被重写为kafka.serializer.StringEncoder 17/06/16 17:52:05 INFO Producer：关闭制片人

更新 kafka是kafka_2.11-0.10.2.0

Answer 1

这是一个警告，您的应用程序不会因此而崩溃。这里的代码正在尝试从MANIFEST.MF加载Kafka版本。我的预感是你使用阴影罐而且它不包括Kafka的MANIFEST.MF。

有关详细信息，请参阅https://issues.apache.org/jira/browse/KAFKA-1901。

scala kafka制作人：无法从MANIFEST.MF读取Kafka版本。可能的原因：java.lang.NullPointerException

1 个答案: