Spark结构化流中的指标以获取总批处理时间

Question

默认情况下，在Spark结构化流式Spark中，打印指标，但根据文档 triggerExecution ，必须是执行整个批次所花费的总时间。

triggerExecution = batch_time = (addBatch + getBatch + getEndOffset + queryPlanning  + setOffsetRange + triggerExecution + walCommit)

不是这种情况，总批处理时间也不完全等于 triggerExecution 时间。 批处理的总执行时间与

batch_time = triggerExecution + addBatch

指标

"timestamp" : "2020-10-06T09:45:41.721Z",
  "batchId" : 12609,
  "numInputRows" : 199800,
  "inputRowsPerSecond" : 651.0624570275969,
  "processedRowsPerSecond" : 517.4758161639968,
  "durationMs" : {
    "addBatch" : 357041,
    "getBatch" : 0,
    "getEndOffset" : 1,
    "queryPlanning" : 29,
    "setOffsetRange" : 3012,
    "triggerExecution" : 386105,
    "walCommit" : 12949
  }

参考

• https://www.waitingforcode.com/apache-spark-structured-streaming/query-metrics-apache-spark-structured-streaming/read
• https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-ProgressReporter.html