错误kafka010.KafkaRDD:Kafka ConsumerRecord无法序列化。在调用.persist或.window之前,使用.map提取字段

时间:2019-06-18 09:17:38

标签: spark-streaming-kafka

我使用的是yarn and cluster模式,streaming-kafka版本是0.10。然而, 经常出现以下错误“ ERROR kafka010.KafkaRDD:Kafka ConsumerRecord无法序列化。使用.map提取字段,然后调用.persist或.window”吗?是什么原因?

val topic = Configuration.new_feed
    val groupId = "star_feed_consumer"
    val duration = "1000"
    val kafkaParams = KafkaUtil.initKafkaParams(Configuration.bootstrap_servers_log, groupId, duration)
    val topics = Array(topic)
    val stream = KafkaUtils.createDirectStream(ssc, PreferConsistent, Subscribe[String, String](topics, kafkaParams))
    val cachedStream = stream.cache()
    val new_feeds = cachedStream.map(record => parseJson(record.value))

    //new_feeds.foreachRDD(rdd => println(rdd))
    new_feeds.print(10)

0 个答案:

没有答案