Question

我目前正在these instructions之后在Amazon EC2 linux VM上构建Apache Spark。

我用于建筑的工具：

apache-maven：3.2.5;

scala：2.10.4;
     锌：0.3.5.3;

Java：jdk1.7.0_79

Linux 32bits

引发此错误消息：

Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) on project spark-core_2.10: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile failed. CompileFailed -> [Help 1]

[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 
[ERROR] Re-run Maven using the -X switch to enable full debug logging. 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

网站建议错误可能是由插件失败引起的，但没有提供详细信息。问题是什么？有没有办法解决错误？

Answer 1

您可以使用以下pom.xml来构建您的项目

<properties>
  <spark.version>2.3.2</spark.version>
  <scala.version>2.11.12</scala.version>
  <scala.compat.version>2.11</scala.compat.version>
</properties>

<dependencies>
  <dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>${scala.version}</version>
    <scope>provided</scope>
  </dependency>
  <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_${scala.compat.version}</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
  </dependency>
  <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_${scala.compat.version}</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
  </dependency>
  <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_${scala.compat.version}</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
  </dependency>
</dependencies>
<build>
  <sourceDirectory>src/main/scala</sourceDirectory>
  <testSourceDirectory>src/test/scala</testSourceDirectory>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-assembly-plugin</artifactId>
      <configuration>
        <archive>
          <manifest>
            <mainClass>package.name.of.main.object</mainClass> <!-- add the path to file containing main method e.g com.company.code.ObjectName -->
          </manifest>
        </archive>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
      </configuration>
      <executions>
        <execution>
          <id>make-assembly</id>
          <phase>package</phase>
          <goals>
            <goal>single</goal>
          </goals>
        </execution>
      </executions>
    </plugin>

    <plugin>
      <groupId>net.alchim31.maven</groupId>
      <artifactId>scala-maven-plugin</artifactId>
      <version>3.1.0</version>
      <executions>
        <execution>
          <!-- <phase>compile</phase> -->
          <goals>
            <goal>compile</goal>
            <goal>testCompile</goal>
          </goals>
        </execution>
      </executions>
    </plugin>

    <plugin>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.3</version>
      <configuration>
        <source>1.8</source>
        <target>1.8</target>
      </configuration>
    </plugin>
</build>

在包含pom文件的目录中

运行命令：mvn clean install，您的项目将以uber / fat jar的形式出现在目标目录中。然后，您可以照常通过JAR进行火花提交。

请记住以下几点：

Spark和Scala不支持Java 1.7。如果您想使用最新的Spark 2.x系列，您必须在您的Scala 2.11 / 2.12中使用依赖项。
如果您使用的是Spark 1.6，则最好使用Scala 2.11，因为该支持对于Scala 2.10，将不会那么容易获得。

基于Amazon EC2的Apache Spark构建错误，Spark-Maven-Plugin失败

1 个答案: