Question

在尝试简化Spark和Scala的单元测试时，我使用的是scala-test和mockito-scala（以及mockito糖）。这只是让您执行以下操作：

val sparkSessionMock = mock[SparkSession]

然后，您通常可以使用“何时”和“验证”来完成所有魔术。

但是如果您有一些实现必须导入的实现

import spark.implicits._

在其代码中，单元测试的简单性似乎消失了（或者至少我还没有找到解决该问题的最合适方法）。

我最终遇到此错误：

org.mockito.exceptions.verification.SmartNullPointerException: 
You have a NullPointerException here:
-> at ...
because this method call was *not* stubbed correctly:
-> at scala.Option.orElse(Option.scala:289)
sparkSession.implicits();

由于键入问题，仅模拟SparkSession中“隐式”对象的调用无济于事：

val implicitsMock = mock[SQLImplicits]
when(sparkSessionMock.implicits).thenReturn(implicitsMock)

不允许您通过，因为它说这将需要您的模拟对象的类型：

require: sparkSessionMock.implicits.type
found: implicitsMock.type

请不要告诉我，我应该执行SparkSession.builder.getOrCreate（）...，因为那样的话，这不再是单元测试，而是更重的集成测试。

（编辑）：这是一个完整的可复制示例：

import org.apache.spark.sql._
import org.mockito.Mockito.when
import org.scalatest.{ FlatSpec, Matchers }
import org.scalatestplus.mockito.MockitoSugar

case class MyData(key: String, value: String)

class ClassToTest()(implicit spark: SparkSession) {
    import spark.implicits._

    def read(path: String): Dataset[MyData] = 
         spark.read.parquet(path).as[MyData]
}

class SparkMock extends FlatSpec with Matchers with MockitoSugar {

     it should "be able to mock spark.implicits" in {
         implicit val sparkMock: SparkSession = mock[SparkSession]
         val implicitsMock = mock[SQLImplicits]
         when(sparkMock.implicits).thenReturn(implicitsMock)
         val readerMock = mock[DataFrameReader]
         when(sparkMock.read).thenReturn(readerMock)
         val dataFrameMock = mock[DataFrame]
         when(readerMock.parquet("/some/path")).thenReturn(dataFrameMock)
         val dataSetMock = mock[Dataset[MyData]]
         implicit val testEncoder: Encoder[MyData] = Encoders.product[MyData]
         when(dataFrameMock.as[MyData]).thenReturn(dataSetMock)

         new ClassToTest().read("/some/path/") shouldBe dataSetMock
    }
 }

Answer 1

您不能模拟隐式。隐式在编译时解决，而模拟在运行时发生（运行时反射，通过 Byte Buddy）。您不能在编译时导入仅在运行时模拟的隐式。您必须手动解析隐式（原则上，如果您在运行时再次启动编译器，则可以在运行时解析隐式，但这会更加困难1 2 3 {{3} }。

尝试

class ClassToTest()(implicit spark: SparkSession, encoder: Encoder[MyData]) {
  def read(path: String): Dataset[MyData] = 
    spark.read.parquet(path).as[MyData]
}

class SparkMock extends AnyFlatSpec with Matchers with MockitoSugar {

  it should "be able to mock spark.implicits" in {
    implicit val sparkMock: SparkSession = mock[SparkSession]
    val readerMock = mock[DataFrameReader]
    when(sparkMock.read).thenReturn(readerMock)
    val dataFrameMock = mock[DataFrame]
    when(readerMock.parquet("/some/path")).thenReturn(dataFrameMock)
    val dataSetMock = mock[Dataset[MyData]]
    implicit val testEncoder: Encoder[MyData] = Encoders.product[MyData]
    when(dataFrameMock.as[MyData]).thenReturn(dataSetMock)

    new ClassToTest().read("/some/path") shouldBe dataSetMock
  }
}

//[info] SparkMock:
//[info] - should be able to mock spark.implicits
//[info] Run completed in 2 seconds, 727 milliseconds.
//[info] Total number of tests run: 1
//[info] Suites: completed 1, aborted 0
//[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
//[info] All tests passed.

请注意，"/some/path"在两个地方都应该相同。在您的代码段中，两个字符串是不同的。

Spark Scala模拟spark.implicit用于单元测试

1 个答案: