Question

我在Ubuntu 14.04 LTS上安装了 Spark 1.5 。使用命令build/mvn -Dscala-2.11 -DskipTests clean package运行构建时，我在项目Spark SQL期间遇到以下构建错误：

    [error] missing or invalid dependency detected while loading class file 'WebUI.class'.
        [error] Could not access term eclipse in package org,
        [error] because it (or its dependencies) are missing. Check your build definition for
        [error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
        [error] A full rebuild may help if 'WebUI.class' was compiled against an incompatible version of org.
        [error] missing or invalid dependency detected while loading class file 'WebUI.class'.
        [error] Could not access term jetty in value org.eclipse,
        [error] because it (or its dependencies) are missing. Check your build definition for
        [error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
        [error] A full rebuild may help if 'WebUI.class' was compiled against an incompatible version of org.eclipse.
        [warn] 22 warnings found
        [error] two errors found
        [error] Compile failed at Sep 18, 2015 6:09:38 PM [17.330s]
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] 
        [INFO] Spark Project Parent POM ........................... SUCCESS [  6.723 s]
        [INFO] Spark Project Core ................................. SUCCESS [03:07 min]
    ...
        [INFO] Spark Project Catalyst ............................. SUCCESS [ 58.166 s]
        [INFO] Spark Project SQL .................................. FAILURE [ 19.912 s]
        [INFO] Spark Project Hive ................................. SKIPPED
        [INFO] Spark Project Unsafe ............................... SKIPPED
...
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------

下面是我在档案.bashrc

中的env变量

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64

export SCALA_HOME=/usr/local/src/scala/scala-2.11.7
export PATH=$SCALA_HOME/bin:$PATH
export PATH=/home/ubuntu/apache-maven-3.3.3/bin:$PATH

export SPARK_HOME=/home/ubuntu/spark-1.5.0
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

更新：尝试使用-Ylog-classpath运行，但不起作用：

Unable to parse command line options: Unrecognized option: -Ylog-classpath

Answer 1

只需从spark目录运行./dev/change-scala-version.sh 2.11即可将所有代码切换为2.11。然后在设置了标志的情况下运行mvn（3.3.3+）或make-distribution.sh。

Answer 2

请参阅Angelo Genovese的评论，不要在构建命令中包含-Dscala-2.11。

Answer 3

如果你不需要spark-sql，那么只需从build中排除与sql相关的模块：

mvn clean package -Dscala-2.11 -DskipTests -pl '!sql/core,!sql/catalyst,!sql/hive'

Answer 4

我也遇到了这个问题，在我从Maven pom.xml导入IntelliJ的项目中。我的同事帮助我弄清楚虽然<scope>runtime</scope>对于大多数依赖项是可行的，但这个特定的依赖关系需要<scope>compile</scope>（由于我们不理解的原因）：

<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-reflect</artifactId>
    <version>${scala.version}</version>
    <scope>compile</scope>
</dependency>

Answer 5

通过运行位于@ spark-1.6.1 / dev / change-scala-version.sh的“change-scala-version.sh”命令，首先将scala版本从2.10更改为2.11，可以解决此构建问题。 / p>

有关详细信息，请参阅以下链接。 http://gibbons.org.uk/spark-on-windows-feb-2016

Spark SQL 1.5构建失败

5 个答案: