我已经在Spark Streaming应用程序中配置了jmx指标。以下是代码:
val sc = spark.sparkContext
spark.conf.set("spark.sql.streaming.metricsEnabled", "true")
spark.conf.set(s"spark.metrics.conf.*.sink.jmx.class", "org.apache.spark.metrics.sink.JmxSink")
UserMetricsSystem.initialize(sc, config.getAppNamespace)
val listener = new EventCollector(isSingleStream) // Some custom code
spark.streams.addListener(listener)
具有metrics.properties的内容是:
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
配置完所有这些并运行应用程序后,我可以在以下位置看到应用程序的inputRate,延迟和processingRate:
jconsole <host>:<port> // driver host
但是我想在浏览器中以json格式查看这些指标。 有没有办法通过Spark API访问这些已配置的jmx指标?