想要监控Kafka消息传递指标 - 主要是消耗或丢失了多少消息，延迟，最终消费者抵消等等......
找到Apache Spark，但我真的不明白，如果它可以“轻松插入Kafka”而不编写任何（java）代码。
我希望在一些配置文件中更改Kafka连接的配置，在哪里存储邮件传递指标并在某些UI中显示和查询。
使用Spark-Kafka消息传递监控是否有“更自动化”的方式，而不是从头开始编写（java）应用程序？
是否有一些已经完成的应用程序，可以自由使用？
如何以最简单和自动化的方式设置链Kafka - Spark - （数据库？） - UI？
Spark还可以监控其他指标 - 集群，代理，主题，不兼容的副本，分区数量......等等吗？
答案 0 :(得分：0)
您好，可以使用kafka manager项目。它将以方便的方式管理所有kafka服务
答案 1 :(得分：0)
使用不同的软件进行实验后，kafka-monitor显示最适合我的情况。
它通过整个集群创建分区主题，并提供有关生产和消费的指标，如消息丢失，重复和端到端滞后。
像这样启动kafka-monitor：
cd <path_to_kafka_monitor>
export JMX_PORT=<some_port>
./bin/kafka-monitor-start.sh config/kafka-monitor.properties
使用jmxTrans收集指标。使用的查询：
{
"servers": [{
"port": "9999",
"host": "127.0.0.1",
"queries": [{
"obj": "kafka.consumer:type=consumer-coordinator-metrics,client-id=*",
"attr": ["assigned-partitions","commit-latency-avg","commit-rate","heartbeat-rate","heartbeat-response-time-max",
"join-rate","join-time-avg","join-time-max","last-heartbeat-seconds-ago","sync-rate","sync-time-avg","sync-time-max"],
"resultAlias": "monitor-consumer.consumer-coordinator-metrics;client-id",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id"]
}]
},
{
"obj": "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*",
"attr": ["bytes-consumed-rate","fetch-latency-avg","fetch-latency-max","records-consumed-rate","records-lag-max",
"records-per-request-avg"],
"resultAlias": "monitor-consumer.consumer-fetch-manager-metrics,client-id",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id"]
}]
},
{
"obj": "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*",
"attr": ["bytes-consumed-rate","fetch-size-avg","fetch-size-max","records-consumed-rate","records-per-request-avg"],
"resultAlias": "monitor-consumer-fetch-manager-metrics;client-id,topic",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id","topic"]
}]
},
{
"obj": "kafka.consumer:type=consumer-metrics,client-id=*",
"attr": [],
"resultAlias": "monitor-consumer-metrics;client-id",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id"]
}]
},
{
"obj": "kafka.consumer:type=consumer-node-metrics,client-id=*,node-id=*",
"attr": ["incoming-byte-rate","outgoing-byte-rate","request-latency-avg","request-rate","request-size-avg","response-rate"],
"resultAlias": "monitor-consumer-node-metrics;client-id,node-id",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id","node-id"]
}]
},
{
"obj": "kafka.producer:type=producer-metrics,client-id=*",
"attr": [],
"resultAlias": "monitor-producer-metrics;client-id",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id"]
}]
},
{
"obj": "kafka.producer:type=producer-node-metrics,client-id=*,node-id=*",
"attr": ["incoming-byte-rate","outgoing-byte-rate","request-latency-avg","request-rate","request-size-avg","response-rate"],
"resultAlias": "monitor-producer-node-metrics;client-id,node-id",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id","node-id"]
}]
},
{
"obj": "kafka.producer:type=producer-topic-metrics,client-id=*,topic=*",
"attr": [],
"resultAlias": "monitor-producer-topic-metrics;client-id,topic",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor",
"typeNames": ["client-id","topic"]
}]
},
{
"obj": "kmf.services:type=consume-service,name=single-cluster-monitor",
"attr": [],
"resultAlias": "monitor-consume-service.single-cluster-monitor",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor"
}]
},
{
"obj": "kmf.services:type=produce-service,name=single-cluster-monitor",
"attr": [],
"resultAlias": "monitor-produce-service.single-cluster-monitor",
"outputWriters": [{
"@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
"url": "http://127.0.0.1:8086/",
"username": "root",
"password": "root",
"database": "monitor"
}]
}
]
}]
}