Question

以下是我的代码排序方式，

//Accumulator initialized 
val count = new LongAccumulator
sparksession.sparkContext.register(count,"count accumulator")

// Streaming Transformation
val DF = fromKafkaDF.map{
  count.add(1)
  println(count.value)  // This value is one
  //some transformation
}.writeStream.outputMode("update").format("console").start()

//trying to access the value of accumulator from driver
println(count.value)  //this value is zero

为什么驱动程序中累加器的值为零？我有其他逻辑可以基于这个累加器工作。请建议。

Answer 1

要让您累加计数器中的任何值，您应该执行一个操作并检查，如果不执行任何操作就不会使用累加器，请检查链接accumulator explained

Answer 2

在喜欢的听众中使用它，

class TestListner(acc: LongAccumulator) extends StreamingQueryListener {
  override def onQueryStarted(event: StreamingQueryListener.QueryStartedEvent): Unit = {
    println("onQueryStarted   :" + event.toString)
  }

  override def onQueryProgress(event: StreamingQueryListener.QueryProgressEvent): Unit = {
    println(acc)
    acc.reset()
    println("onQueryProgress    :" + event.progress)
  }

  override def onQueryTerminated(event: StreamingQueryListener.QueryTerminatedEvent): Unit = {
    println("onQueryProgress    :" + event)
  }
}

和主应用程序中的addListener

    spark.streams.addListener(new TestListner(acc))

如何在驱动程序中提取累加器值？

2 个答案: