从external.api获取查询时,Prometheus Adapter Goroutine问题

时间:2019-10-05 01:24:43

标签: kubernetes monitoring grafana prometheus

我已经实现了外部api并将其注册到k8s环境,但是在获取指标时遇到了一些问题。

这是外部api指标的规则:

  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        kubernetes_namespace: {resource: "namespace"}
        kubernetes_pod_name: {resource: "pod"}
    name:
      matches: "^(.*)_total"
      as: "${1}"

检查kubernetes外部api指标时,资源值不是null,看起来不错,但

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "http_requests",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

完成所有这些操作后,当我想要检查返回此错误的http_requests度量kubernetes的值时;

Error from server (InternalError): Internal error occurred: unable to fetch metrics

我已经从prometheus-adapter窗格中检查了此问题,这些是与此查询相关的错误日志;

E1005 01:21:26.387384       1 provider.go:47] unable to generate a query for the metric: empty query produced by metrics query template
I1005 01:21:26.387594       1 wrap.go:42] GET /apis/external.metrics.k8s.io/v1beta1/namespaces/default/http_requests: (504.956µs) 500
goroutine 1909 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420374930, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420374930, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42105b2a0, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42107f920, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:307 +0x45
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1791e20, 0x10, 0x7f340078e6c8, 0xc4204ec580, 0x190fd20, 0xc420e8db88, 0xc421081600, 0x1f4, 0x18f9920, 0xc420039320)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d

....
....
....

但是我已经解析了此错误消息:unable to generate a query for the metric: empty query produced by metrics query template

此问题的原因是什么?

请帮助

1 个答案:

答案 0 :(得分:0)

我认为您收到此错误:

  

无法为指标生成查询:产生的空查询   指标查询模板

因为您的api指标不包含metricsQuery值。

第二件事是,我不确定您为什么在external.metrics中使用http_request。在这种情况下,您可以使用custom.metrics。 提及的指标是最受欢迎的指标之一,因此您可以在线逐步找到许多教程,以了解如何实现这一目标。 例如this或此Github tutorial