__spark_conf__.zip:没有这样的文件或目录

时间:2018-04-21 17:43:27

标签: azure apache-spark hadoop yarn azure-data-factory

当我从HDInsight群集上的Azure数据工厂V2提交Spark活动时,我收到错误 -

Application application_1524230454528_0060 failed 5 times due to AM Container for appattempt_1524230454528_0060_000005 exited with exitCode: -1000
    For more detailed output, check the application tracking page: http://hn1-rbc-cl.tsgjwtochfbexi5wvtaxuipmmg.rx.internal.cloudapp.net:8088/cluster/app/application_1524230454528_0060 Then click on links to logs of each attempt.
    Diagnostics: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    java.io.FileNotFoundException: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2732)
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2644)
    at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
    at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Failing this attempt. Failing the application.

有人可以帮我弄清楚问题是什么,或者有任何其他解决方法通过Azure数据工厂管道运行spark作业。我坚持这个,无法继续。根据Azure Data Factory文档,我尝试将我的文件放在容器中的./files文件夹中,其中包含我的spark jar。但仍面临同样的问题。

1 个答案:

答案 0 :(得分:0)

您的HDI群集版本和用于开发程序的Java版本是什么,请确保版本相同。

BTW,HDI群集3.5+需要java 8,而早期版本需要java 7。