Question

我是Spark Echo系统的新手，试图将数据从CSV写入镶木地板但是因NullPointerException而失败。不确定我错过了什么。

getlisting

引发以下异常。

case class PdRecordData(id: String, dates: String, dayOfWeek: String,      pdDistrict: String,address: String, longitude: String, latitude: String)

val sqlContext = new SQLContext(sc)
sqlContext.createParquetFile[PdRecordData]("C:\\AS\\Parquet", true, new   Configuration()).registerTempTable("PdRegistry")
val csvFile = sc.textFile("C:\\AS\\crimeratedata\\samplefromorg.csv")
val rowsWithHeader = csvFile.map { x => x.split(",").map { _.trim } }
val maps = rowsWithHeader.map { case Array(id,dates, dayOfWeek, pdDistrict, address, longitude, latitude) => PdRecordData(id,dates, dayOfWeek, pdDistrict, address, longitude, latitude) }
maps.foreach { x => sqlContext.sql("INSERT INTO PdRegistry SELECT " +  "'" + x.dates + "','" + x.dayOfWeek + "','" + x.pdDistrict + "','" + x.address + "','" + x.longitude + "','" + x.latitude + "'")}

Answer 1

试试这个

val df = sqlContext.createDataFrame(maps)
sqlContext.write.parquet(path)

sql conf nullpointer实木复合地板

1 个答案: