我使用的是yarn and cluster模式,streaming-kafka版本是0.10。然而, 经常出现以下错误“ ERROR kafka010.KafkaRDD:Kafka ConsumerRecord无法序列化。使用.map提取字段,然后调用.persist或.window”吗?是什么原因?
val topic = Configuration.new_feed
val groupId = "star_feed_consumer"
val duration = "1000"
val kafkaParams = KafkaUtil.initKafkaParams(Configuration.bootstrap_servers_log, groupId, duration)
val topics = Array(topic)
val stream = KafkaUtils.createDirectStream(ssc, PreferConsistent, Subscribe[String, String](topics, kafkaParams))
val cachedStream = stream.cache()
val new_feeds = cachedStream.map(record => parseJson(record.value))
//new_feeds.foreachRDD(rdd => println(rdd))
new_feeds.print(10)