org.apache.spark.SparkException:写入行

时间:2018-02-28 19:32:25

标签: apache-spark apache-spark-sql apache-spark-dataset

您好我收到以下代码,试图将DataFrame保存到提供的路径

df.write.mode(SaveMode.Append).parquet("/path/year=2018")

当我使用上面的代码时,我得到以下异常。我正在使用Spark 2.2.0。我是Spark的新手。

  

org.apache.spark.SparkException:在写入行时任务失败       org.apache.spark.sql.execution.datasources.FileFormatWriter $ $ .ORG阿帕奇$ $火花SQL $ $执行数据源$ $$ FileFormatWriter executeTask(FileFormatWriter.scala:270)       在       org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $写$ 1 $$ anonfun $ $应用MCV $ SP $ 1.适用(FileFormatWriter.scala:189)       在       org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $写$ 1 $$ anonfun $ $应用MCV $ SP $ 1.适用(FileFormatWriter.scala:188)       在org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)at at       org.apache.spark.scheduler.Task.run(Task.scala:108)at       org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:338)   在       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)       在       java.util.concurrent.ThreadPoolExecutor中的$ Worker.run(ThreadPoolExecutor.java:617)       在java.lang.Thread.run(Thread.java:745)

     

引起:java.lang.NullPointerException at           org.apache.spark.sql.SparkSession $$ anonfun $ 3.apply(SparkSession.scala:468)           在           org.apache.spark.sql.SparkSession $$ anonfun $ 3.apply(SparkSession.scala:468)           在scala.collection.Iterator $$ anon $ 11.next(Iterator.scala:409)at           scala.collection.Iterator $$ anon $ 11.next(Iterator.scala:409)at           org.apache.spark.sql.execution.datasources.FileFormatWriter $ SingleDirectoryWriteTask.execute(FileFormatWriter.scala:324)           在           org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $ $组织阿帕奇$火花$ SQL $执行$ $的数据源$$ FileFormatWriter $ executeTask 3.apply(FileFormatWriter.scala:256)           在           org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $ $组织阿帕奇$火花$ SQL $执行$ $的数据源$$ FileFormatWriter $ executeTask 3.apply(FileFormatWriter.scala:254)           在           org.apache.spark.util.Utils $ .tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1371)           在           org.apache.spark.sql.execution.datasources.FileFormatWriter $ $ .ORG阿帕奇$ $火花SQL $ $执行数据源$ $$ FileFormatWriter executeTask(FileFormatWriter.scala:259)

0 个答案:

没有答案