Question

我使用Kafka Avro Producer将avro数据写入kafka主题，现在我想使用Spark Streaming读取该数据。我使用directKafkaStream API读取avro数据，但失败并出现错误。

这是我的火花流代码，正在读取avro数据：

    public static void main(String[] args) throws InterruptedException {

    SparkConf sparkConf = new SparkConf().setAppName("Kafka Streaming").setMaster("local[*]");
    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(2));

    Set<String> topicsSet = new HashSet(Arrays.asList("test_topic"));
    Map<String, String> kafkaParams = new HashMap();
    //kafkaParams.put("metadata.broker.list", "localhost:9092");

    kafkaParams.put("bootstrap.servers", "localhost:9092");
    kafkaParams.put("key.deserializer", StringDeserializer.class.getName());
    kafkaParams.put("value.deserializer", KafkaAvroDeserializer.class.getName());
    kafkaParams.put("schema.registry.url", "http://localhost:7788");
    kafkaParams.put("specific.avro.reader", "true");

    // Create direct kafka stream with brokers and topics
    JavaPairInputDStream<String, Customer> directKafkaStream = KafkaUtils.createDirectStream(
        jssc,
        String.class,
        Customer.class,
        KafkaAvroDecoder.class,
        KafkaAvroDecoder.class,
        kafkaParams,
        topicsSet
    );

    directKafkaStream.print();
    jssc.start();
    jssc.awaitTermination();}

“客户”是我在使用Kafka Avro Producer将avro数据写入test_topic时使用的架构类。

当我启动此代码时，它失败并显示错误：

Error:(38, 78) java: no suitable method found for createDirectStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.Class<java.lang.String>,java.lang.Class<com.example.Customer>,java.lang.Class<io.confluent.kafka.serializers.KafkaAvroDecoder>,java.lang.Class<io.confluent.kafka.serializers.KafkaAvroDecoder>,java.util.Map<java.lang.String,java.lang.String>,java.util.Set<java.lang.String>)
method org.apache.spark.streaming.kafka.KafkaUtils.<K,V,KD,VD>createDirectStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.Class<K>,java.lang.Class<V>,java.lang.Class<KD>,java.lang.Class<VD>,java.util.Map<java.lang.String,java.lang.String>,java.util.Set<java.lang.String>) is not applicable
  (inferred type does not conform to equality constraint(s)
    inferred: java.lang.Object
    equality constraints(s): java.lang.Object,java.lang.String)
method org.apache.spark.streaming.kafka.KafkaUtils.<K,V,KD,VD,R>createDirectStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.Class<K>,java.lang.Class<V>,java.lang.Class<KD>,java.lang.Class<VD>,java.lang.Class<R>,java.util.Map<java.lang.String,java.lang.String>,java.util.Map<kafka.common.TopicAndPartition,java.lang.Long>,org.apache.spark.api.java.function.Function<kafka.message.MessageAndMetadata<K,V>,R>) is not applicable
  (cannot infer type-variable(s) K,V,KD,VD,R
    (actual and formal argument lists differ in length))
method org.apache.spark.streaming.kafka.KafkaUtils.<K,V,KD,VD>createDirectStream(org.apache.spark.streaming.StreamingContext,scala.collection.immutable.Map<java.lang.String,java.lang.String>,scala.collection.immutable.Set<java.lang.String>,scala.reflect.ClassTag<K>,scala.reflect.ClassTag<V>,scala.reflect.ClassTag<KD>,scala.reflect.ClassTag<VD>) is not applicable
  (cannot infer type-variable(s) K,V,KD,VD
    (argument mismatch; org.apache.spark.streaming.api.java.JavaStreamingContext cannot be converted to org.apache.spark.streaming.StreamingContext))
method org.apache.spark.streaming.kafka.KafkaUtils.<K,V,KD,VD,R>createDirectStream(org.apache.spark.streaming.StreamingContext,scala.collection.immutable.Map<java.lang.String,java.lang.String>,scala.collection.immutable.Map<kafka.common.TopicAndPartition,java.lang.Object>,scala.Function1<kafka.message.MessageAndMetadata<K,V>,R>,scala.reflect.ClassTag<K>,scala.reflect.ClassTag<V>,scala.reflect.ClassTag<KD>,scala.reflect.ClassTag<VD>,scala.reflect.ClassTag<R>) is not applicable
  (cannot infer type-variable(s) K,V,KD,VD,R
    (actual and formal argument lists differ in length))

当我使用StringDecoder而不是KafkaAvroDecoder时，相同的代码可以工作，但是接收到的数据格式不可读（有些疯狂的字符），但是我希望在接收到数据时对数据进行解码，以便使我对数据有所了解。请提出正确的做法。

无法在Spark Streaming中使用直接Kafka流反序列化Avro数据

0 个答案: