Question

我试图让Spark 1.4.1在IntelliJ Idea 14.1中与Scala 2.11.7一起使用，但我继续收到此错误：

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the `Hadoop binaries.`

有人知道我需要下载哪些二进制文件吗？

来自spark的我的pom.xml就是这个（开始）：

    <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.apache</groupId>
    <artifactId>apache</artifactId>
    <version>14</version>
  </parent>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-parent_2.10</artifactId>
  <version>1.4.1</version>
  <packaging>pom</packaging>
  <name>Spark Project Parent POM</name>
  <url>http://spark.apache.org/</url>
  <licenses>
    <license>
      <name>Apache 2.0 License</name>
      <url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
      <distribution>repo</distribution>
    </license>
  </licenses>
  <scm>
    <connection>scm:git:git@github.com:apache/spark.git</connection>
    <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/spark.git</developerConnection>
    <url>scm:git:git@github.com:apache/spark.git</url>
    <tag>HEAD</tag>
  </scm>
  <developers>
    <developer>
      <id>matei</id>
      <name>Matei Zaharia</name>
      <email>matei.zaharia@gmail.com</email>
      <url>http://www.cs.berkeley.edu/~matei</url>
      <organization>Apache Software Foundation</organization>
      <organizationUrl>http://spark.apache.org</organizationUrl>
    </developer>
  </developers>
  <issueManagement>
    <system>JIRA</system>
    <url>https://issues.apache.org/jira/browse/SPARK</url>
  </issueManagement>

  <prerequisites>
    <maven>3.0.4</maven>
  </prerequisites>

降级到Scala 2.10.5会更好吗？

Answer 1

我的设置完全相同。看看

https://issues.apache.org/jira/browse/SPARK-2356?focusedCommentId=14395185&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14395185

可以从HortonWorks服务器下载。将winutils.exe放在某个文件夹中，并在Scala程序中以下列方式引用它：

System.setProperty("hadoop.home.dir", "D:\\YOUR_PATH_TO_WINUTILS")

IntelliJ Idea 14.1中Spark 1.4.1的winutils二进制文件

1 个答案: