Question

我对Amazon的AWS上的Kubernetes和Prometheus的自定义指标有疑问。默认情况下，CPU和内存指标运行良好。普罗米修斯不是http_requests，这是错误消息：

$ kubectl describe hpa hpa-deploy
Name:                       hpa-deploy
Namespace:                  default
Labels:                     <none>
Annotations:                kubectl.kubernetes.io/last-applied-configuration:
                              {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-deploy","namespace":"default...
CreationTimestamp:          Thu, 06 Jun 2019 11:06:48 +0000
Reference:                  Deployment/django
Metrics:                    ( current / target )
  "http_requests" on pods:  <unknown> / 2k
Min replicas:               1
Max replicas:               10
Deployment pods:            1 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server could not find the metric http_requests for pods
Events:
  Type     Reason               Age                     From                       Message
  ----     ------               ----                    ----                       -------
  Warning  FailedGetPodsMetric  8m53s (x414 over 114m)  horizontal-pod-autoscaler  unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
  Warning  FailedGetPodsMetric  3m48s (x12 over 6m36s)  horizontal-pod-autoscaler  unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server could not find the metric http_requests for pods

我使用github项目建议的头盔来安装Prometheus，并检查了api：

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

然后添加了以下规则：

$ kubectl edit cm my-release-prometheus-adapter
    rules:
    - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

walkthrough说，在添加新规则后，返回的api检查应该在“ resources”：[]内包含值，但是没有，我也不知道为什么。

这是我的hpa代码：

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-deploy
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: django
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests
        target:
          type: Value
          averageValue: 2k

此外，我使用的是基于Nginx的Ingress Controller，但是Ingress和Service的hpa kubectl描述表明：

$ kubectl describe hpa hpa-ingress
Name:                                                      hpa-ingress
Namespace:                                                 default
Labels:                                                    <none>
Annotations:                                               kubectl.kubernetes.io/last-applied-configuration:
                                                             {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-ingress","namespace":"defaul...
CreationTimestamp:                                         Thu, 06 Jun 2019 11:06:48 +0000
Reference:                                                 Ingress/test-ingress
Metrics:                                                   ( current / target )
  "http_requests" on Ingress/test-ingress (target value):  <unknown> / 2k
Min replicas:                                              1
Max replicas:                                              10
Ingress pods:                                              0 current / 0 desired
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: the server could not find the requested resource
Events:
  Type     Reason          Age                     From                       Message
  ----     ------          ----                    ----                       -------
  Warning  FailedGetScale  2m40s (x473 over 122m)  horizontal-pod-autoscaler  the server could not find the requested resource

我不确定是否必须手动导出Pod的http_requests指标，如果是这种情况，该怎么办？文档全部都是“复制粘贴，一切都会正常”，但事实并非如此。请，如果可能的话，越详细越好，这是我的新话题。非常感谢。

Answer 1

就我而言，这是由于错误的普罗米修斯端点。我通过将日志级别设置为 6 发现了这一点，并发现 prometheus 查询的日志因 404 错误而失败。

https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/README.md#why-isnt-my-metric-showing-up

Prometheus的http_request自定义指标在Kubernetes中不起作用

1 个答案: