Question

我正在使用play JSON库工具来解析Spark中的JSON数据，并收到以下错误消息。有没有人对这个错误的可能原因有任何疑问？如果这是由于错误的JSON记录，我该如何识别坏记录？谢谢！

以下是我用来解析JSON数据的主要脚本：

import play.api.libs.json._
val jsonData = distdata.map(line => Json.parse(line)) //line 194 of script parseJson_v14.scala
val filteredData = jsonData.map(json => (json \ "QueryStringParameters" \ "pr").asOpt[String].orNull).countByValue()

变量distdata是一种文本格式的JSON数据，变量jsonData是JsValue数据的rdd。由于Spark转换是惰性的，因此在执行第二个命令创建变量filteredData之前错误没有跳出，并且根据错误消息，错误来自我创建变量jsonData的第一个命令。

[2017-03-29 14:55:39.616]-[Logging$class.logWarning]-[WARN]: Lost task 42.0 in stage 1.0 (TID 90, 10.119.126.114): com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input at [Source: ; line: 1, column: 1] at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148) at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3110) at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3024) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1652) at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:226) at play.api.libs.json.Json$.parse(Json.scala:21) at parseJson_v14$$anonfun$1$$anonfun$3$$anonfun$apply$1.apply(parseJson_v14.scala:194) at parseJson_v14$$anonfun$1$$anonfun$3$$anonfun$apply$1.apply(parseJson_v14.scala:194) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$6.apply$mcV$sp(PairRDDFunctions.scala:1197) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$6.apply(PairRDDFunctions.scala:1197) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$6.apply(PairRDDFunctions.scala:1197) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1250) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1205) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Answer 1

检查distdata中是否没有空行，并且您在一行中有所有JSON对象，例如

{"id":"121", "name":"robot 1"}
{"id":"122", "name":"robot 2"}

与

相反

{"id":"121", "name":
"robot 1"}
{"id":"122", "name":
"robot 2"}

解析json时，由于输入结束而没有要映射的内容

1 个答案: