我们解决了发送过多次警报的问题。 我们使用一个实例的普罗米修斯和一个实例的警报管理器（无群集）。
我们希望收到来自Alertmanager的消息，其中服务器已关闭（警报处于启动状态），服务器又可用（警报处于已解决状态）。 尤其有趣的是，当所有服务器又都可用时（警报组处于已解决状态）。
当一个或多个服务器再次可用时，我们会收到多个呼叫，并带有不同的已解决警报。 我们应该只收到一次警报已解决的信息。 我们仅通过消息收到一个警报（即使两个警报已解决，请参阅最后一个操作）。
global:
resolve_timeout: 8737h
route:
group_by: ['alertname']
group_wait: 0s
group_interval: 60s
repeat_interval: 8737h
receiver: 'web.hook'
receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://localhost:9991/v1/alerts/'
send_resolved: true
请注意：
我们不需要在重复间隔内解决超时问题，因此我们使用此处建议的较大值：＃935）
global:
scrape_interval: 5s
evaluation_interval: 20s
scrape_configs:
- job_name: 5c5d8e41c680703f3a26fb68-monitoring-1
scrape_interval: 60s
scrape_timeout: 10s
metrics_path: /metrics
static_configs:
- targets:
- server01
- server02
- server03
- server04
- server05
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 'http-scenario-exporter:9900'
rule_files:
- rules/5c5d8e41c680703f3a26fb68-monitoring-1.yml
groups:
- name: 5c5d8e41c680703f3a26fb68-monitoring-1
rules:
- alert: 5c5d8e41c680703f3a26fb68-monitoring-1-0
expr: >-
http_scenario_ok{job='5c5d8e41c680703f3a26fb68-monitoring-1'}
== 0
for: 60s
labels: {}
annotations: {}
这是我们重现此问题的方式（请注意，instanceName标记由导出程序设置）。
server01，server02，server03，server04，server05已启动。
我停止server05：
{
"level": 10,
"time": "2019-02-11T13:51:25.895Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
我停止server04：
{
"level": 10,
"time": "2019-02-11T13:57:45.994Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
我重新启动server5：
{
"level": 10,
"time": "2019-02-11T14:01:25.998Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
我停止了server03：（在这里我收到两个呼叫（！）：一个呼叫server04 + server03触发，另一个呼叫server04 + server03触发且server05解决！）
{
"level": 10,
"time": "2019-02-11T14:03:46.001Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:04:06.002Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
我重新启动server04：（在这里，我收到很多呼叫（！）。总是在server03触发时启动。 有时解决了server04，有时解决了server05。）
{
"level": 10,
"time": "2019-02-11T14:07:46.002Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:08:06.004Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:09:06.005Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:09:26.004Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1",
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:10:26.005Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:11:46.005Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1",
},
"_commonAnnotations": {
},
"v": 1
}
...