提交火花申请时的错误

时间:2016-09-01 13:55:32

标签: java hadoop apache-spark cluster-computing apache-spark-standalone

我正在尝试提交一个非常简单的应用程序,它包括从一个输入大文件(大约500 GO)创建两个rdds,减去标题(第一行),用索引压缩它们,将它们映射到键值一个小修改,然后将它们保存为文本文件

我能够在spark web UI上看到作业的进度,最后2个作业由于此错误而失败请告诉我它是什么原因以及如何解决它

  

作业因阶段失败而中止:阶段2.0中的任务4897失败1次,最近失败:阶段2.0中失去的任务4897.0(TID 4914,localhost):java.io.IOException:Aucun espace disponible surlepériphérique       at java.io.FileOutputStream.writeBytes(Native Method)       在java.io.FileOutputStream.write(FileOutputStream.java:345)       在org.apache.spark.storage.TimeTrackingOutputStream.write(TimeTrackingOutputStream.java:58)       在java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)       在java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)       at net.jpountz.lz4.LZ4BlockOutputStream.finish(LZ4BlockOutputStream.java:243)       at net.jpountz.lz4.LZ4BlockOutputStream.close(LZ4BlockOutputStream.java:175)       在org.apache.spark.serializer.DummySerializerInstance $ 1.close(DummySerializerInstance.java:65)       at org.apache.spark.storage.DiskBlockObjectWriter $$ anonfun $ close $ 2.apply $ mcV $ sp(DiskBlockObjectWriter.scala:108)       在org.apache.spark.util.Utils $ .tryWithSafeFinally(Utils.scala:1296)       在org.apache.spark.storage.DiskBlockObjectWriter.close(DiskBlockObjectWriter.scala:107)       at org.apache.spark.storage.DiskBlockObjectWriter.commitAndClose(DiskBlockObjectWriter.scala:132)       在org.apache.spark.shuffle.sort.ShuffleExternalSorter.writeSortedFile(ShuffleExternalSorter.java:188)       在org.apache.spark.shuffle.sort.ShuffleExternalSorter.closeAndGetSpills(ShuffleExternalSorter.java:410)       在org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:204)       at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:164)       在org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)       在org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)       在org.apache.spark.scheduler.Task.run(Task.scala:85)       在org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:274)       在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)       at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:615)       在java.lang.Thread.run(Thread.java:745)

驱动程序堆栈跟踪:

PS;有一句法语句子:

  

Aucun espace disponible surlepériphérique:这意味着此设备上没有可用空间,

我认为这是原因,但我不明白哪种设备以及如何解决它

0 个答案:

没有答案