Question

在Dataproc群集中启动的火花作业失败，但以下异常。我尝试了各种群集配置，但结果是相同的。我在Dataproc映像1.2中遇到此错误。

注意：没有抢占式工作器，磁盘中也有足够的空间。但是，我注意到工作节点上根本没有/hadoop/yarn/nm-local-dir/usercache/root文件夹。但是我可以看到一个名为dr.who的文件夹。

java.io.IOException: Failed to create local dir in /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1534256335401_0001/blockmgr-89931abb-470c-4eb2-95a3-8f8bfe5334d7/2f.
    at org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:70)
    at org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:80)
    at org.apache.spark.shuffle.IndexShuffleBlockResolver.getDataFile(IndexShuffleBlockResolver.scala:54)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:68)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:86)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

可能的副本：Spark on Google's Dataproc failed due to java.io.FileNotFoundException: /hadoop/yarn/nm-local-dir/usercache/root/appcache/

Answer 1

我可以使用Dataproc 1.3解决此问题。但是1.3并没有附带需要处理的bigquery连接器。 https://cloud.google.com/dataproc/docs/concepts/connectors/bigquery

Dataproc上的Spark失败，并出现java.io.FileNotFoundException：

1 个答案: