我正在运行Hadoop集群HDP-2.2.4.2-2。 我想在Windows机器上开发MapReduce作业。我已经使用Hadoop插件配置了Eclipse,Eclipse中的HDFS浏览器工作正常。但是,在群集上启动基本WordCount作业后,它最终会出现错误:
Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:730)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:488)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:528)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:506)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:298)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
at wordcount.WordCount.main(WordCount.java:102)
我也尝试在发生错误之前调试和变量如下:
http://postimg.org/image/73ubc44ev/
看起来像文件系统错误 - eclipse尝试使用linux chmod更改Windows文件系统文件的权限...但我不确定它是否是NullPointerException的原因。还有som null变量可能导致这个问题 - 目录和环境。
有什么想法吗?感谢