在k8s自动缩放器中使用Prometheus自定义指标

Question

我正在尝试从k8s应用程序公开自定义指标，以供Horizo​​ntal Pod自动缩放器使用。指标是整个应用程序范围的，而不是每个实例的（主要是，积压工作队列的长度，对于所有工作程序实例都相同，并且来自共享后端），因此我将应用程序分为两个部署：k8worker是“ worker bee”实例可以自动缩放的指标，而k8worker-metrics是提供指标的独立部署。拆分的目的是维护数量有限的k8worker-metrics实例，因此，如果k8worker扩展到30个实例，Prometheus不必调用30次即可获得完全相同的指标。我打算将k8worker-metrics实例的数量保持为2或3（但为了保持弹性，不能超过1个）。

但是我无法将指标集成到HPA中。

我最初尝试根据从k8worker-metrics容器中刮取的指标建立HPA清单。 我可以看到来自custom.metrics.k8s.io的指标读数：

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
....
    {
      "name": "pods/k8worker_work_queue_length",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
....
  ]
}

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/k8worker_work_queue_length" | jq .

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/k8worker_work_queue_length"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "k8worker-metrics-758656df7-x67mh",
        "apiVersion": "/v1"
      },
      "metricName": "k8worker_work_queue_length",
      "timestamp": "2020-03-03T04:06:47Z",
      "value": "8"
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "k8worker-metrics-758656df7-58dxb",
        "apiVersion": "/v1"
      },
      "metricName": "k8worker_work_queue_length",
      "timestamp": "2020-03-03T04:06:47Z",
      "value": "8"
    }
  ]
}

但是，如果我在HPA清单中声明对度量的引用为

  - type: Object
    object:
      metric:
        name: k8worker_work_queue_length
      describedObject:
        apiVersion: custom.metrics.k8s.io/v1beta1
        kind: Pod
        name: k8worker-metrics
      target:
        type: Value
        value: 4

然后我收到一条消息

$ kubectl describe hpa
....
"FailedGetObjectMetric  ... unable to get metric k8worker_work_queue_length: Pod on default k8worker-metrics/unable to fetch metrics from custom metrics API: unable to map kind Pod.custom.metrics.k8s.io to resource: no matches for kind "Pod" in group "custom.metrics.k8s.io"

我还尝试将k8worker-metrics容器包装为服务（k8worker-metrics-svc），希望我可以在HPA清单中将其引用为

 - type: Object
   object:
     metric:
       name: k8worker_work_queue_length
     describedObject:
       apiVersion: custom.metrics.k8s.io/v1beta1
       kind: Service
       name: k8worker-metrics-svc
     target:
       type: Value
       value: 4

但是服务读数未显示在/apis/custom.metrics.k8s.io中：

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep queue_length
      "name": "namespaces/k8worker_work_queue_length",
      "name": "pods/k8worker_work_queue_length",
      "name": "jobs.batch/k8worker_work_queue_length",

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/k8worker-metrics-svc/k8worker_work_queue_length"
Error from server (NotFound): the server could not find the metric k8worker_work_queue_length for services

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep serv
[empty output]

即使该服务及其读数在Prometheus UI中可见（要捕获以下屏幕截图，我都将Pod和该服务都标注为抓取）

非常感谢您的建议。