在Windows

时间:2017-05-29 18:15:12

标签: hadoop apache-spark intellij-14

spark Intellij上运行简单Windows程序时获取异常。当我在调试模式下运行代码时,发现程序尝试执行以下命令失败。

D:\winutils\bin\winutils.exe chmod 0644 C:\Users\himanshu\git\SparkRDDs\target\output\_temporary\0\_temporary\attempt_20170529130949_0027_m_000000_405\part-00000

这是代码失败的地方。

join.rdd.saveAsTextFile("file:///Users/himanshu/git/SparkRDDs/target/output/")

下面的Stacktrace -

  
    

17/05/29 14:07:48错误执行者:第27.0阶段任务0.0中的例外(TID 410)         org.apache.hadoop.util.Shell $ ExitCodeException:             在org.apache.hadoop.util.Shell.runCommand(Shell.java:464)             在org.apache.hadoop.util.Shell.run(Shell.java:379)             在org.apache.hadoop.util.Shell $ ShellCommandExecutor.execute(Shell.java:589)             在org.apache.hadoop.util.Shell.execCommand(Shell.java:678)             在org.apache.hadoop.util.Shell.execCommand(Shell.java:661)             at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)             在org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)             在org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)             在org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)             在org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)             在org.apache.hadoop.fs.FileSystem.create(FileSystem.java:798)             at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)             在org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)             在org.apache.spark.rdd.PairRDDFunctions $$ anonfun $ saveAsHadoopDataset $ 1 $$ anonfun $ 13.apply(PairRDDFunctions.scala:1206)             在org.apache.spark.rdd.PairRDDFunctions $$ anonfun $ saveAsHadoopDataset $ 1 $$ anonfun $ 13.apply(PairRDDFunctions.scala:1197)             在org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)             在org.apache.spark.scheduler.Task.run(Task.scala:99)             在org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:322)             在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)             at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:617)             在java.lang.Thread.run(Thread.java:745)

  
Surprisingly in Eclipse, the program runs just fine.

0 个答案:

没有答案