Spark JSON DStream Print()/ saveAsTextFiles不起作用

时间:2016-07-30 18:42:24

标签: scala printing spark-streaming dstream

Issue Description:

Spark Version: 1.6.2
Execution: Spark-shell (REPL) master = local[2] (tried local[*])

example.json is as below:

{"name":"D2" ,"lovesPandas":"Y"}
{"name":"D3" ,"lovesPandas":"Y"}
{"name":"D4" ,"lovesPandas":"Y"}
{"name":"D5" ,"lovesPandas":"Y"} 

在Spark-shell本地模式下执行代码:

import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._ 
import org.apache.spark.streaming.kafka._
import org.apache.spark.sql._
import org.json4s._
import org.json4s.jackson.JsonMethods._
import _root_.kafka.serializer.StringDecoder
import _root_.kafka.serializer.Decoder
import _root_.kafka.utils.VerifiableProperties
import org.apache.hadoop.hbase._
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.client.Put
import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.hbase.mapred.TableOutputFormat
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.mapred.JobConf
import org.apache.kafka.clients.consumer.ConsumerConfig
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.StreamingContext

val ssc = new  StreamingContext(sc,  Seconds(2) )
val messages = ssc.textFileStream("C:\\pdtemp\\test\\example.json")

messages.print()

I tried the saveAsTextFiles but it is not saving any files too.

这不起作用 - 显示没有输出 - 尝试在火花壳上读取Kafka的流

也尝试了以下方法 - 不起作用:

messages.foreachRDD(rdd => rdd.foreach(print))

此外,尝试解析模式转换为数据帧但似乎没有任何工作

正常的json解析工作正常,我可以将正常// RDD / DF //的内容打印到Spark-shell中的控制台

有人可以帮忙吗?

0 个答案:

没有答案