Question

我试图将Kube状态指标部署到运行Kubernetes v1.14的EKS集群（eks.4）的kube-system名称空间中。

Kubernetes连接

provider "kubernetes" {
  host                   = var.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
  cluster_ca_certificate = base64decode(var.cluster.certificate)
  load_config_file       = true
}

部署清单（如.tf）

resource "kubernetes_deployment" "kube_state_metrics" {
  metadata {
    name      = "kube-state-metrics"
    namespace = "kube-system"

    labels = {
      k8s-app = "kube-state-metrics"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        k8s-app = "kube-state-metrics"
      }
    }

    template {
      metadata {
        labels = {
          k8s-app = "kube-state-metrics"
        }
      }

      spec {
        container {
          name  = "kube-state-metrics"
          image = "quay.io/coreos/kube-state-metrics:v1.7.2"

          port {
            name           = "http-metrics"
            container_port = 8080
          }

          port {
            name           = "telemetry"
            container_port = 8081
          }

          liveness_probe {
            http_get {
              path = "/healthz"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }

          readiness_probe {
            http_get {
              path = "/"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }
        }

        service_account_name = "kube-state-metrics"
      }
    }
  }
}

我还从https://github.com/kubernetes/kube-state-metrics/tree/master/kubernetes部署了所有必需的RBAC清单-为简洁起见，在此进行了编辑。

当我在上面的部署中运行terraform apply时，Terraform输出如下： kubernetes_deployment.kube_state_metrics: Still creating... [6m50s elapsed]

最终在10m超时。

以下是kube-state-metrics窗格的日志输出

I0910 23:41:19.412496       1 main.go:140] metric white-blacklisting: blacklisting the following items:
W0910 23:41:19.412535       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0910 23:41:19.412565       1 client_config.go:546] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
F0910 23:41:19.412782       1 main.go:148] Failed to create client: invalid configuration: no configuration has been provided

Answer 1

我没有尝试使用Terraform。

我刚刚在本地运行此部署，却遇到了相同的错误。

请在本地运行部署以查看部署和Pod的状态。

I0910 13:25:49.632847       1 main.go:140] metric white-blacklisting: blacklisting the following items:
W0910 13:25:49.632871       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

 and finally:

I0910 13:25:49.634748       1 main.go:185] Testing communication with server
I0910 13:25:49.650994       1 main.go:190] Running with Kubernetes cluster version: v1.12+. git version: v1.12.8-gke.10. git tree state: clean. commit: f53039cc1e5295eed20969a4f10fb6ad99461e37. platform: linux/amd64
I0910 13:25:49.651028       1 main.go:192] Communication with server successful
I0910 13:25:49.651598       1 builder.go:126] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses
I0910 13:25:49.651607       1 main.go:226] Starting metrics server: 0.0.0.0:8080
I0910 13:25:49.652149       1 main.go:201] Starting kube-state-metrics self metrics server: 0.0.0.0:8081

验证：

Connected to kube-state-metrics (xx.xx.xx.xx) port 8080 (#0)
 GET /metrics HTTP/1.1
 Host: kube-state-metrics:8080
 User-Agent: curl/7.58.0
 Accept: */*

HTTP/1.1 200 OK
 Content-Type: text/plain; version=0.0.4
 Date: Tue, 10 Sep 2019 13:39:52 GMT
 Transfer-Encoding: chunked

 [49027 bytes data]
 HELP kube_certificatesigningrequest_labels Kubernetes labels converted to 
Prometheus labels.

如果您要建立自己的形象，请关注gihtub和docs上的问题

更新：只是为了澄清。

我在回答中提到。我没有尝试使用terraform，但似乎第一个问题仅描述了一个问题W0910 13:25:49.632871 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

因此，我建议在本地运行此部署，并从日志中验证所有错误。后来发生automount_service_account_token问题。这个重要错误并未应用于原始问题。因此，请遵循terraform issues on github 您如何设法解决此问题

根据description on github：

我花了数小时试图弄清为什么服务帐户和部署不能在Terraform中工作，但是在kubectl中却没有任何问题-这是因为AutomountServiceAccountToken在部署资源中被硬编码为False。

至少应该在Terraform文档中记录该资源，并注意该资源的行为不像kubectl。

我希望它能解释这个问题。

Answer 2

在spec中添加以下内容已使我成功部署。

automount_service_account_token = true

后代：

resource "kubernetes_deployment" "kube_state_metrics" {
  metadata {
    name      = "kube-state-metrics"
    namespace = "kube-system"

    labels = {
      k8s-app = "kube-state-metrics"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        k8s-app = "kube-state-metrics"
      }
    }

    template {
      metadata {
        labels = {
          k8s-app = "kube-state-metrics"
        }
      }

      spec {
        automount_service_account_token = true
        container {
          name  = "kube-state-metrics"
          image = "quay.io/coreos/kube-state-metrics:v1.7.2"

          port {
            name           = "http-metrics"
            container_port = 8080
          }

          port {
            name           = "telemetry"
            container_port = 8081
          }

          liveness_probe {
            http_get {
              path = "/healthz"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }

          readiness_probe {
            http_get {
              path = "/"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }
        }

        service_account_name = "kube-state-metrics"
      }
    }
  }
}

EKS kube系统部署CrashLoopBackOff

2 个答案: