Question

我的hdfs中有orc文件。字段之一是Map（String，String）。不知何故，在此字段中有一些行的值为Map（null，null）。映射键中的null是Java的严重错误。因此，当我尝试访问该字段时，我得到了NullPointer证明。

我想读取这些文件并将此字段更改为emty map。

我试图这样做：

val df = spark.read.format("orc").load("/tmp/bad_orc")

def func(s: org.apache.spark.sql.Row): String = { 
    try
    {
        if ( s(14) == null ) // the 14'th column is the column with Map(String,String) type
        {
            return "Ok"
        }
        else
        {
            return "Zero"
        }
    }
    catch
    {
        case x: Exception => return "Erro"
    }
}
df.rdd.map(func).take(20)

我在运行此脚本时遇到了这个异常。

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 417.0 failed 4 times, most recent failure: Lost task 0.3 in stage 417.0 (TID 97094, srvg1076.local.odkl.ru, executor 86): java.lang.NullPointerException
    at java.util.TreeMap.compare(TreeMap.java:1294)
    at java.util.TreeMap.put(TreeMap.java:538)
    at org.apache.orc.mapred.OrcMapredRecordReader.nextMap(OrcMapredRecordReader.java:507)
    at org.apache.orc.mapred.OrcMapredRecordReader.nextValue(OrcMapredRecordReader.java:554)
    at org.apache.orc.mapreduce.OrcMapreduceRecordReader.nextKeyValue(OrcMapreduceRecordReader.java:104)
    at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)

当我尝试访问该orc中的任何其他列时-一切正常。

如何捕获此异常以及如何修复这些文件？请帮助我

读取带有损坏的地图的Spark ORC文件

0 个答案: