Question

我需要为Kubernetes集群组件（apiserver，apiserver-kubelet-client，apiserver-etcd-client，front-proxy-client等）生成自己的SSL证书。原因是因为默认情况下这些证书的有效期设置为1年，并且由于我的业务原因，我需要将有效期设置为一年以上。当我生成自己的一组证书并初始化了集群时，一切都正常运行-kube系统名称空间中的POD启动，与apiserver的通信正常。但我感到有些惊奇，例如kubectl logs或kubectl port-forward或kubectl exec之类的命令停止工作并开始引发以下错误：

kubectl logs <kube-apiserver-pod> -n kube-system
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <kube-apiserver-pod>))

或

kubectl exec -it <kube-apiserver-pod> -n kube-system sh
error: unable to upgrade connection: Unauthorized`

但是docker exec命令登录到k8s_apiserver容器是否正常工作。

在调试过程中，我发现只有自行生成的apiserver-kubelet-client密钥/证书文件才导致此群集行为。

下面是我用来为apiserver-kubelet-client生成并使用自己的证书/密钥对的过程。

我将kubernetes集群初始化为通过运行/etc/kubernetes/pki

kubeadm init ...

将/etc/kubernetes/pki文件夹备份到/tmp/pki_k8s
使用openssl打开apiserver-kubelet-client.crt，以检查所有设置的扩展区，CN，O等。

openssl x509 -noout -text -in /tmp/pki_k8s/apiserver-kubelet-client.crt
为确保相同的扩展名和CN，O参数出现在我自己生成的证书中，我为扩展名创建了.conf文件，为CN和O创建了.csr文件

cd /tmp/pki_k8s/ cat <<-EOF_api_kubelet_client-ext > apiserver_kubelet_client-ext.conf [ v3_ca ] keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth EOF_api_kubelet_client-ext

openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"
最后，我生成了自己的apiserver-kubelet-client.crt。对于这一代，我重用了由K8S初始化生成的apiserver-kubelet-client.key和ca.crt / ca.key

openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -out apiserver-kubelet-client.crt -extensions v3_ca -extfile apiserver_kubelet_client-ext.conf -days 3650
一旦我生成了自己的apiserver-kubelet-client.crt来覆盖由k8s初始化脚本本身生成的上一个脚本，我就可以通过点击kubeadm reset来重置kubernetes集群。这清除了/ etc / kubernetes文件夹
将所有证书从/etc/kubernetes/pki复制到/tmp/pki_k8s
并重新初始化K8S群集kubeadm init ...

在此期间，我看到K8S集群使用了/etc/kubernetes/pki中存储的现有证书进行设置。

[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Using the existing front-proxy-ca certificate and key.
[certificates] Using the existing front-proxy-client certificate and key.
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

在那之后，K8S集群启动，我可以列出pod，列出说明，进行部署等，但是如上所述无法检查日志和exec命令。

 kubectl get pods -n kube-system
NAME                                           READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-kjkp9                       1/1       Running   0          2m
coredns-78fcdf6894-q88lx                       1/1       Running   0          2m
...

kubectl  logs <apiserver_pod> -n kube-system -v 7
I0818 08:51:12.435494   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.436355   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.438413   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.447751   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.448109   12811 round_trippers.go:383] GET https://<HOST_IP>:6443/api/v1/namespaces/kube-system/pods/<apiserver_pod>
I0818 08:51:12.448126   12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.448135   12811 round_trippers.go:393]     Accept: application/json, */*
I0818 08:51:12.448144   12811 round_trippers.go:393]     User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.462931   12811 round_trippers.go:408] Response Status: 200 OK in 14 milliseconds
I0818 08:51:12.471316   12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.471949   12811 round_trippers.go:383] GET https://<HOST_IP>:6443/api/v1/namespaces/kube-system/pods/<apiserver_pod>/log
I0818 08:51:12.471968   12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.471977   12811 round_trippers.go:393]     Accept: application/json, */*
I0818 08:51:12.471985   12811 round_trippers.go:393]     User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.475827   12811 round_trippers.go:408] Response Status: 401 Unauthorized in 3 milliseconds
I0818 08:51:12.476288   12811 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials ( pods/log <apiserver_pod>)",
  "reason": "Unauthorized",
  "details": {
    "name": "<apiserver_pod>",
    "kind": "pods/log"
  },
  "code": 401
}]
F0818 08:51:12.476325   12811 helpers.go:119] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <apiserver_pod>))

请参阅下面的kubelet服务文件：

[root@qa053 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="CA_CLIENT_CERT=--client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELE=--rotate-certificates=true"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CERTIFICATE_ARGS $CA_CLIENT_CERT

您有什么想法吗？ :) 谢谢

最好的问候

Answer 1

我找出了它不起作用的原因。

创建.csr文件时，我使用了以下方法：

openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"

但是-subj中的格式错误，导致从证书解析权限CN时出现问题。不需要"/O=system:masters,CN=kube-apiserver-kubelet-client"，而是

openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters/CN=kube-apiserver-kubelet-client"

两个.csr文件生成的证书在-text视图方面看起来相同。但是他们的行为有所不同。

使用自行生成的apiserver-kubelet-client.crt证书初始化K8S集群时，某些kubectl命令不起作用

1 个答案: