Question

我正在使用群集方案。如果我将一个文件考虑到hdfs中，但是当我尝试将所有文件列入目录时，代码运行良好，以便每次处理一个文件，如下所示

val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val files = new File("hdfs://nh-me:9000/user/user1/TotalFIle/").listFiles.map(_.getName).toList

我收到以下错误，请注意出现错误的第53行是我发布的代码段中的第四行：

17/02/14 23:57:19 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NullPointerException
java.lang.NullPointerException
at     scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:192)
at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:192)
at scala.collection.SeqLike$class.size(SeqLike.scala:106)
at scala.collection.mutable.ArrayOps$ofRef.size(ArrayOps.scala:186)
at scala.collection.mutable.Builder$class.sizeHint(Builder.scala:69)
at scala.collection.mutable.ArrayBuilder.sizeHint(ArrayBuilder.scala:22)
at scala.collection.TraversableLike$class.builder$1(TraversableLike.scala:230)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at SparkGraphGen$.main(CorrDistGraph.scala:53)
at SparkGraphGen.main(CorrDistGraph.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627)
17/02/14 23:57:19 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NullPointerException)

Answer 1

您无法使用Java File API访问HDFS上的文件。以下是使用HDFS API访问文件的示例：

import org.apache.hadoop.fs._
import org.apache.hadoop.conf.Configuration

val path = new Path("hdfs://nh-me:9000/user/user1/TotalFIle/")
val fs = path.getFileSystem(new Configuration())
val files = fs.listStatus(path).map(_.getPath.getName).toList

当我尝试将文件发现到HDFS时，NullPointerException，spark

1 个答案: