Question

我正在使用外部Nginx负载平衡器并尝试配置K8s Master，但失败并出现以下错误：

错误上传配置：无法创建configmap：禁止configmaps：用户“ system：anonymous”无法在名称空间“ kube-system” **中创建configmaps

对我来说，这更像是证书问题，但是我很难找到我所缺少的东西，我们在基础架构中使用F5负载均衡器在apiserver之前提供了任何帮助，在这里我看到了同样的问题我为故障排除而创建的环境

kubeadm-config：

    apiVersion: kubeadm.k8s.io/v1alpha2
    kind: MasterConfiguration
    kubernetesVersion: v1.11.0
    apiServerCertSANs:
    - "ec2-23-23-244-63.compute-1.amazonaws.com"
    api:
    controlPlaneEndpoint: "ec2-23-23-244-63.compute-1.amazonaws.com:6443"
    etcd:
    external:
        endpoints:
        - https://172.31.32.160:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
     networking:
     # This CIDR is a calico default. Substitute or remove for your CNI provider.
     podSubnet: "10.244.0.0/16"

环境： Kubelet：1.11.1 kubeadm 1.11.1 kubectl 1.11.1

输出

    [certificates] Using the existing ca certificate and key.
    [certificates] Using the existing apiserver certificate and key.
    [certificates] Using the existing apiserver-kubelet-client certificate and 
    key.
    [certificates] Using the existing sa key.
    [certificates] Using the existing front-proxy-ca certificate and key.
    [certificates] Using the existing front-proxy-client certificate and key.
    [certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
     [endpoint] WARNING: port specified in api.controlPlaneEndpoint overrides 
  api.bindPort in the controlplane address
     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller- 
  manager.conf"




     [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
     [controlplane] wrote Static Pod manifest for component kube-apiserver to 
  "/etc/kubernetes/manifests/kube-apiserver.yaml"
     [controlplane] wrote Static Pod manifest for component kube-controller-manager 
  to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
     [controlplane] wrote Static Pod manifest for component kube-scheduler to 
  "/etc/kubernetes/manifests/kube-scheduler.yaml"
     [init] waiting for the kubelet to boot up the control plane as Static Pods 
  from directory "/etc/kubernetes/manifests"
  [init] this might take a minute or longer if the control plane images have to 
  be pulled
     [apiclient] All control plane components are healthy after 41.036802 seconds


     [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" 
     in the "kube-system" Namespace
     error uploading configuration: unable to create configmap: configmaps is 
     forbidden: User "system:anonymous" cannot create configmaps in the 
     namespace "kube-system"

日志：

    Unable to register node "ip-172-31-40-157" with API server: nodes is forbidden: User "system:anonymous" cannot create nodes at the cluster scope
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
on_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "ip-172-31-40-157" not found
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
:172] Unable to update cni config: No networks found in /etc/cni/net.d

Nginx：

   upstream mywebapp1 {
      server 172.31.40.157:6443;

   }
server {
    listen 6443 ssl;
    server_name ec2-23-23-244-63.compute-1.amazonaws.com;

    ssl on;
    ssl_certificate         /opt/certificates/server.crt;
    ssl_certificate_key     /opt/certificates/server.key;
    ssl_trusted_certificate /opt/certificates/ca.crt;

    location / {
        proxy_pass https://mywebapp1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Nginx Server : 172-31-44-203
Master Server : 172-31-40-157

我正在使用自签名证书和CA来生成所有证书，包括nginx中的证书都是相同的

使用f5负载均衡器

Answer 1

如果您的节点通过负载平衡器与apiserver对话，并且希望使用客户端证书凭据进行身份验证（这是节点的典型做法），则负载平衡器不得终止或重新加密TLS，否则客户端证书信息将丢失，并且apiserver将请求视为匿名请求。

Answer 2

Nginx 负载均衡主节点配置：

stream {
    upstream stream_backend {
        least_conn;
        server master01:6443;
        server master02:6433;
        server master03:6433;
    }

    server {
        listen        6443;
        proxy_pass    stream_backend;
        proxy_timeout 3s;
        proxy_connect_timeout 1s;
    }
}

使用外部Nginx负载平衡器或F5负载平衡器的Kubernetes HA安装程序用户“ system：anonymous”无法在集群范围内创建节点

2 个答案: