Question

我想了解如何测试相同的火花代码。谷歌搜索我找到了spark-tetsting-base。好吧，现在我想尝试一下，但我无法用Maven运行它。

首先，我有可用mvn package打包的scala代码，并具有以下项目结构：

pom.xml
src/main/scala/com/test/spark/mycode.scala
src/test/scala/com/test/spark/test.scala

问题是当我运行mvn test时，它没有在src/main/com/test/spark/test.scala下运行测试。该测试实际上是spark-testing-base wiki的第一个例子。 maven的输出是：

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ mycode ---
[INFO] Nothing to compile - all classes are up to date

我的pom.xml看起来像这样：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.test.spark</groupId>
  <artifactId>mycode</artifactId>
  <version>0.0.1</version>
  <name>${project.artifactId}</name>
  <description>Simple wtest</description>
  <inceptionYear>2017</inceptionYear>

  <!-- change from 1.6 to 1.7 depending on Java version -->
  <properties>
    <maven.compiler.source>1.6</maven.compiler.source>
    <maven.compiler.target>1.6</maven.compiler.target>
    <encoding>UTF-8</encoding>
    <scala.version>2.11.5</scala.version>
    <scala.compat.version>2.11</scala.compat.version>
    <spark.version>1.6.1</spark.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>

    <!-- Spark dependency -->
    <dependency>         
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- Spark sql dependency -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- Spark hive dependency -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- spark-testing-base dependency -->
    <dependency>
        <groupId>com.holdenkarau</groupId>
        <artifactId>spark-testing-base_2.11</artifactId>
        <version>${spark.version}_0.6.0</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.scalactic</groupId>
        <artifactId>scalactic_2.11</artifactId>
        <version>3.0.1</version>
    </dependency>
    <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_2.11</artifactId>
        <version>3.2.0-SNAP5</version>
        <scope>test</scope>
    </dependency>
 </dependencies>

  <build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>
    <!-- Create JAR with all dependencies -->
    <plugins>
       <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
      </plugin>
      <plugin>
        <!-- see http://davidb.github.com/scala-maven-plugin -->
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.2.2</version>
        <configuration>
            <scalaVersion>${scala.version}</scalaVersion>
            <scalaCompatVersion>${scala.compat.version}</scalaCompatVersion>
        </configuration>
        <executions>
          <execution>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <!-- for testing scala code -->
      <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.20</version>
          <configuration>
            <argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

任何帮助表示赞赏。谢谢！

修改1：

我将其添加为test.scala以便测试失败：

class test extends FunSuite with SharedSparkContext {
  test("test initializing spark context") {
    val list = List(1, 2, 3, 4)
    val rdd = sc.parallelize(list)
    val list2 = List(1, 2, 3, 4, 5)

    assert(rdd.count === list2.length)
  }
}

但我仍然得到：

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ mycode ---
[INFO] Nothing to compile - all classes are up to date
[INFO]

[INFO] --- maven-surefire-plugin:2.20:test (default-test) @ mycode ---
[INFO]------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]------------------------------------------------------------------------

尽管assert中的比较是假的，好像测试没有运行一样。：（

编辑2： 我用这个插件改变了pom，仍然没有改变：

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.20</version>
  <configuration>
    <includes>
      <include>**/*Test.scala</include>
    </includes>
    <argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine>
  </configuration>
</plugin>

编辑3： 添加<goal>testCompile</goal>到net.alchim31.maven以编译测试类。尽管测试仍然没有运行。最终让测试运行的缺失是什么？

编辑4： 感谢您的意见。我将pom更改为以下内容：

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

<groupId>com.test.spark</groupId>
<artifactId>testJavaAndScala</artifactId>
<version>1.0-SNAPSHOT</version>
<name>Test for Java + Scala compilation</name>
<description>Test for Java + Scala compilation</description>
<inceptionYear>2017</inceptionYear>

<properties>
    <scala.version>2.11.5</scala.version>
    <scala.compat.version>2.11</scala.compat.version>
    <spark.version>1.6.1</spark.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
    </dependency>
    <!-- Spark dependency -->
    <dependency>         
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- Spark sql dependency -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- Spark hive dependency -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- Spark-testing-base -->
    <dependency>
        <groupId>com.holdenkarau</groupId>
        <artifactId>spark-testing-base_${scala.compat.version}</artifactId>
        <version>${spark.version}_0.6.0</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.1</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.0.2</version>
            </plugin>
        </plugins>
    </pluginManagement>
    <plugins>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <executions>
                <execution>
                    <id>scala-compile-first</id>
                    <phase>process-resources</phase>
                    <goals>
                        <goal>add-source</goal>
                        <goal>compile</goal>
                    </goals>
                </execution>
                <execution>
                    <id>scala-test-compile</id>
                    <phase>process-test-resources</phase>
                    <goals>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <executions>
                <execution>
                    <phase>compile</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
    </build>

</project>

这是mvn test输出：

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Test for Java + Scala compilation 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ testJavaAndScala ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/src/main/resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.1:add-source (scala-compile-first) @ testJavaAndScala ---
[INFO] Add Source directory: /home/src/main/scala
[INFO] Add Test Source directory: /home/src/test/scala
[INFO]
[INFO] --- scala-maven-plugin:3.2.1:compile (scala-compile-first) @ testJavaAndScala ---
[INFO] /home/src/main/scala:-1: info: compiling
[INFO] Compiling 1 source files to /home/target/classes at 1497938635103
[INFO] prepare-compile in 0 s
[INFO] compile in 3 s
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ testJavaAndScala ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:compile (default) @ testJavaAndScala ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ testJavaAndScala ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- scala-maven-plugin:3.2.1:testCompile (scala-test-compile) @ testJavaAndScala ---
[INFO] /home/src/test/scala:-1: info: compiling
[INFO] Compiling 1 source files to /home/target/test-classes at 1497938639022
[INFO] prepare-compile in 0 s
[INFO] compile in 5 s
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ testJavaAndScala ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ testJavaAndScala ---
[INFO] Surefire report directory: /home/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.854 s
[INFO] Finished at: 2017-06-20T08:04:05+02:00
[INFO] Final Memory: 28M/360M
[INFO] ------------------------------------------------------------------------

我可以在目标目录中找到我的测试类，但似乎没有执行。这是为什么？正如我所说，它只是最小的例子而且它不依赖于我的实际主代码。

Answer 1

Surefire不会发现它自己的那些最大的测试。你需要使用scalatest-maven-plugin。说明可用on their website

要开始使用，请像这样修改您的pom。

<!-- disable surefire -- >
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.7</version>
  <configuration>
    <skipTests>true</skipTests>
  </configuration>
</plugin>
<!-- enable scalatest -- >
<plugin>
  <groupId>org.scalatest</groupId>
  <artifactId>scalatest-maven-plugin</artifactId>
  <version>1.0</version>
  <configuration>
    <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
    <junitxml>.</junitxml>
    <filereports>WDF TestSuite.txt</filereports>
  </configuration>
  <executions>
    <execution>
      <id>test</id>
      <goals>
        <goal>test</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Answer 2

由于文件夹结构，无法运行测试。你的文件夹结构

src/main/scala/com/test/spark/mycode.scala
src/main/test/com/test/spark/test.scala

文件夹结构应该像

src/main/scala/com/test/spark/mycode.scala
src/test/scala/com/test/spark/test.scala

Here是标准的maven布局。

希望这有帮助！

<强>被修改

您在pom文件中缺少测试执行插件，请参阅scala maven plugin

请在net.alchim31.maven插件

中添加以下内容

            <execution>
                <id>scala-test-compile</id>
                <phase>process-test-resources</phase>
                <goals>
                    <goal>testCompile</goal>
                </goals>
            </execution>

Answer 3

即使这是一个很晚的答复，但我写这篇文章是希望有人可以从中受益，以防他们的scala 单元测试未被发现。导致考试失败的主要途径之一是无声无息。 如果路径上任何目录的名称中都有空格，则scalatest不会选择它。。重命名此类目录以成功运行单元测试。同样，在您的情况下，您可能需要在禁用surefire插件之后添加sacalatest maven插件。

如何在Maven中使用spark-testing-base？

3 个答案: