Question

我在本地计算机上安装了docker / datastax环境，并尝试使用命令＆＃34; spark-submit＆＃34;执行我的第一个spark应用程序。

应用程序非常简单，经典的单词计数示例。我有文件系统访问权限。

执行的命令是：

sudo /usr/share/dse/bin/dse spark-submit --master dse://172.19.0.8:9042 --verbose  --class it.lslab.spark.WordCount spark.word-count-1.0-SNAPSHOT-jar-with-dependencies.jar

收到的错误是：

Caused by: java.io.FileNotFoundException: File file:/home/username/projects/examples/word_count.text does not exist
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606) ~[hadoop-common-2.7.1.3.jar:na]
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819) ~[hadoop-common-2.7.1.3.jar:na]
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596) ~[hadoop-common-2.7.1.3.jar:na]
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) ~[hadoop-common-2.7.1.3.jar:na]
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140) ~[hadoop-common-2.7.1.3.jar:na]
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341) ~[hadoop-common-2.7.1.3.jar:na]
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) ~[hadoop-common-2.7.1.3.jar:na]
    at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108) ~[hadoop-mapreduce-client-core-2.7.1.3.jar:na]
    at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67) ~[hadoop-mapreduce-client-core-2.7.1.3.jar:na]
    at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:245) ~[spark-core_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]

你能帮我吗？

由于 →

Answer 1

可能所有工作节点上都没有名为'/home/username/projects/examples/word_count.text'的文件。

Spark提交和文件系统错误

1 个答案: