Question

我具有特征 AWSClient ，该特征表明扩展它的类是AWSClient，例如SNSClint，S3Client。

trait AWSClient extends Serializable {
  def resourceType:String
}

此类扩展特性 AWSClient 的类是AWSS3Client

class AWSS3Client extends AWSClient {
  private lazy val s3Client: AmazonS3 = getS3Client("", "", "", Some(""))

  def uploadToS3(data: String): PutObjectResult = {
    s3Client.putObject("bucketName", "key", data)
  }
  override def resourceType: String = "s3"

  def getS3Client(accessKey: String, secretKey: String, region: String, KmsCmkId: Option[String]): AmazonS3 = {...}
}

我有一个类，该类使用AWSClient扩展客户端的实例，DataFrame data 和 sparkSession 作为参数。类的 publishData 方法遍历DataFrame data 并将数据发布到传递的客户端。

class PublishData(publishDataClient: AWSClient,
                       data: DataFrame,
                       sparkSession: SparkSession) extends Serializable {
  def publishData = {
    data.foreachPartition { dataSetPartition => {
      dataSetPartition.foreach(row => {
        val calculatedData: String = someOperation(row);
        PublishDataUtil.publishData(publishDataClient, calculatedData)
      })
    }
    }
  }

  def someOperation(row: Row): String = {
    // some dummy operation operation
    return "testdata"
  }
}

object PublishDataUtil extends Serializable {
  def publishData(publishDataClient: AWSClient, calculatedData: String) = {
    if (publishDataClient.resourceType.equals("s3")) {
      publishDataClient.asInstanceOf[AWSS3Client].uploadToS3(calculatedData)
    }
  }
}

现在可以验证（在PublishData上运行测试用例），我已经创建了测试类 PublishDataTest ，并在其构造函数中传递了模拟的AWSS3Client，即 mockS3Client

class PublishDataTest extends FlatSpec
  with Matchers
  with MockitoSugar
  with SparkHelper
  with BeforeAndAfterEach {

  private val mockedPutObjectResult = Mockito.mock(classOf[PutObjectResult], Mockito.withSettings().serializable())
  private val mockS3Client: AWSS3Client = Mockito.mock(classOf[AWSS3Client], Mockito.withSettings().serializable(SerializableMode.ACROSS_CLASSLOADERS))
  when(mockS3Client.resourceType).thenReturn("s3")
  when(mockS3Client.uploadToS3(ArgumentMatchers.anyString())).thenReturn(mockedPutObjectResult)

  "publishData" should "return nothing" in {
    import sparkSession.implicits._

    val data = Seq(
      (1, 11, "stringData1"),
      (2, 22, "stringData2")
    ).toDF("id1", "id2", "stringData")

    new PublishData(mockS3Client, data, sparkSession).publishData
  }

在运行测试用例时，出现以下异常：

[scalatest] - should return nothing *** FAILED *** (405 milliseconds)
[scalatest]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4, localhost, executor driver): org.mockito.exceptions.base.MockitoSerializationIssue:
[scalatest] A class couldn't be found while deserializing a Mockito mock, you should check your classpath. The error was :
[scalatest]   com.abc.poc.AWSS3Client$MockitoMock$610943276$auxiliary$nz6gzLcw
[scalatest] If you are still unsure what is the reason of this exception, feel free to contact us on the mailing list.
[scalatest]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[scalatest]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

版本：

Scala: 2.11
Spark: 2.2.1
ScalaTest = 2.2.x;
Mockito = 2.22.x;
ScalaMock-ScalaTest = 3.2.x;

完整日志：link

MockitoSerialization问题：反序列化Mockito模拟时找不到类

0 个答案: