Question

嘿我试图通过StreamingListener结束对cloudWatch的一些严格指标

类似的东西：

class MyStreamingListener()
extends StreamingListener{

 override def onBatchCompleted(batchCompleted: StreamingListenerBatchCompleted):Unit={
 val cloudWatch = new AmazonCloudWatchClient(new BasicAWSCredentials(awsAccessKeyId, awsSecretKey))
  cloudWatch.setEndpoint("monitoring.eu-west-1.amazonaws.com")
  val putMetricDataRequest = new PutMetricDataRequest()
    putMetricDataRequest.setNamespace("my-name-space")
    val metricDatum = new MetricDatum().withMetricName("test")
    metricDatum.setValue(batchCompleted.batchInfo.numRecords)
    metricDatum.setUnit(StandardUnit.fromValue("Milliseconds"))
    putMetricDataRequest.getMetricData.add(metricDatum)
    cloudWatch.putMetricData(putMetricDataRequest)
}
}

然后在saprkStreaming中使用它：

val streamingContext: StreamingContext = new StreamingContext(spark.sparkContext, Seconds(2))
  streamingContext.addStreamingListener(new LoadIndexStreamingListener)

  val dstream = KinesisUtils.createStream(
      streamingContext, "this-is-just-a-test", "my-stream", "kinesis.eu-west-1.amazonaws.com",
      "eu-west-1", InitialPositionInStream.LATEST, Seconds(2), StorageLevel.MEMORY_AND_DISK_2)
      .map(byteArray => new String(byteArray))
  dstream.print()
  streamingContext.start()
  streamingContext.awaitTermination()

当我在我的集群（EMR）上使用spark-shell进行一些测试时，它运行正常并且指标已发送到CloudWacth

但是当我将代码打包到带有sbt clean assembly的jar并使用spark submit运行它时，我收到以下错误：

java.lang.NoSuchMethodError: com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(Lcom/amazonaws/services/cloudwatch/model/PutMetricDataRequest;)Lcom/amazonaws/services/cloudwatch/model/PutMetricDataResult;

这是我试过的spark -submit命令：

spark-submit --class com.me.sparkTest.App --master local[4]  --packages org.apache.spark:spark-streaming-kinesis-asl_2.11:2.1.0,com.amazonaws:amazon-kinesis-client:1.7.2 clowdwatch-spark-test-assembly-1.0.jar

使用spark-submit时是什么原因导致它失败？

Answer 1

该错误是由于类的编译与AmazonCloudWatchClient.RotClient.ResClient.RotClient.RotClient.RotClient.RotClient.PartClite.DataMetricData具有不同于EMR群集中可用的运行时库的签名而引起的。

解决方案是创建一个具有以下依赖性的超级jar

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.10.75.1</version>
</dependency>

spark StreamingListener cloudWatch集成

1 个答案: