Question

在使用sc.addFile（rscript）添加R脚本时，遇到以下错误。

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:867)
at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:407)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1386)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1340)
at kafkaStreaming.Stream$.main(Stream.scala:22)
at kafkaStreaming.Stream.main(Stream.scala)

Answer 1

从以下链接下载文件：

https://github.com/srccodes/hadoop-common-2.2.0-bin

在初始化sparkContext之前：

  System.setProperty("hadoop.home.dir", """/path/to/hadoop-common-2.2.0-bin-master""")

使用sc.addFile（）在spark中管道

1 个答案: