Question

我想为我的kubernetes集群创建一个秘密。因此，我编写了以下dummy-secret.yaml文件：

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=

运行kubectl create -f dummy-secret.yaml时，我收到以下消息：

Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret: Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...

不确定为什么会发生。

据我了解，我需要在yaml文件中的data键下对所有值进行编码。所以我做了base64编码，但是kubernetes仍然无法像我期望的那样处理yaml秘密文件。

更新：

我使用此命令在Mac上对data值进行编码：

echo -n 'mega_secret_key' | openssl base64

Answer 1

已经回答了这个问题，但是为了将来参考，不需要使用stringData而不是data字段对字符串进行编码，如下所示：

#secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
stringData:
  API_KEY: "STRING_IN_CLEAR_TEXT"
  API_SECRET: "STRING_IN_CLEAR_TEXT"

Answer 2

我从您的编码数据中获得了解码后的值“ mega_secret_key”和“ really_secret_value1”。似乎它们的编码方式不正确。因此，以正确的方式对数据进行编码：

$ echo "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5Cg==

$ echo "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTEK

然后检查它们是否正确编码：

$ echo "bWVnYV9zZWNyZXRfa2V5Cg==" | base64 -d
mega_secret_key

$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTEK" | base64 -d
really_secret_value1

这样就可以了。现在，在您的dummy-secret.yaml中使用它们：

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5Cg==
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTEK

并运行$ kubectl create -f dummy-secret.yaml。

Answer 3

好像您的错误消息是通过不同的dummy-secret.yaml发生的。

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: af76fsdK_cQ89_Hj1Aq
  API_SECRET: bsdfmkwegwegwe

然后：

$ kubectl create -f s.yaml
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...

如果我使用您的原件，效果很好：

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=

然后：

$ kubectl create -f dummy-secret.yaml
secret/dummy-secret created

我正在使用以下版本：

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Answer 4

当尝试以错误的方式删除换行符（正确的方法是删除后缀“ Cg ==”）时，也会发生这种情况。

我使用了cli中的base64，尽管有一些避免NL的方法，例如

https://superuser.com/questions/1225134/why-does-the-base64-of-a-string-contain-n/1225334

它们在MacOS中不起作用，我发现使用python这样更简单

import base64

data = "abc123!?$*&()'-=@~"

# Standard Base64 Encoding
encodedBytes = base64.b64encode(data.encode("utf-8"))
encodedStr = str(encodedBytes, "utf-8")

或者仍然使用纯基于bash的解决方案：

echo 'secret_to_encode' | tr -d \\n | base64

Answer 5

过一会儿，我想回到这个问题，并留下一个提及官方kubernetes docs的答案：

setHasFixedSize(true)

请特别注意notifyItemRangeRemoved，因为它可以确保您的秘密密钥在解码后不会包含“换行符”。

Answer 6

就我而言，我忘记了转换键的值。可能是这样的事情:)。检查您的值

Answer 7

进行备份

    sudo mkdir backup 
    sudo cp -R /etc/kubernetes backup/
    sudo tar -cvzf backup/pki_backup_`hostname`-`date +%Y%m%d`.tar.gz backup/kubernetes/

查看目录 /etc/kubernetes/

ls -l

total 80
-rw------- 1 root root 5440 Mar  3 13:21 admin.conf
drwxr-xr-x 2 root root 4096 Aug 17  2020 audit-policy
-rw-r--r-- 1 root root  368 Mar  4  2020 calico-config.yml
-rw-r--r-- 1 root root  270 Mar  4  2020 calico-crb.yml
-rw-r--r-- 1 root root  341 Mar  4  2020 calico-cr.yml
-rw-r--r-- 1 root root  147 Mar  4  2020 calico-node-sa.yml
-rw-r--r-- 1 root root 6363 Mar  4  2020 calico-node.yml
-rw------- 1 root root 5472 Mar  3 13:21 controller-manager.conf
-rw-r--r-- 1 root root 3041 Aug 14  2020 kubeadm-config.v1alpha3.yaml
-rw------- 1 root root 5548 Mar  3 13:21 kubelet.conf
-rw-r--r-- 1 root root 1751 Mar  4  2020 kubelet.env
drwxr-xr-x 2 kube root 4096 Aug 14  2020 manifests
lrwxrwxrwx 1 root root   28 Mar  4  2020 node-kubeconfig.yaml -> /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5420 Mar  3 13:21 scheduler.conf
drwxr-xr-x 3 kube root 4096 Mar  3 10:20 ssl

尝试找到 k8s 集群配置，在我的情况下这是 kubeadm-config.v1alpha3.yaml 如果没有相同的文件，您可以生成

kubectl get cm kubeadm-config -n kube-system -o yaml > /etc/kubernetes/kubeadm-config.yaml

在我的情况下也没有文件夹 /etc/kubernetes/pki 并且存在 ~/ssl 我从 /etc/kubernetes/ssl

创建符号链接 /etc/kubernetes/pki

ln -s /etc/kubernetes/ssl /etc/kubernetes/pki

让我们重新生成证书

kubeadm alpha phase certs apiserver  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Using the existing apiserver certificate and key.

kubeadm alpha phase certs apiserver-kubelet-client

I0303 13:12:24.543254   40613 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing apiserver-kubelet-client certificate and key.

kubeadm alpha phase certs front-proxy-client

I0303 13:12:35.660672   40989 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing front-proxy-client certificate and key.

kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [prod-uct1-mvp-k8s-0001 localhost] and IPs [127.0.0.1 ::1]

kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Using the existing etcd/server certificate and key.

kubeadm alpha phase certs  etcd-healthcheck-client --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/healthcheck-client certificate and key.

kubeadm alpha phase certs  etcd-peer --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [product1-mvp-k8s-0001 localhost] and IPs [192.168.4.201 127.0.0.1 ::1]

检查实际证书

find /etc/kubernetes/pki/ -name '*.crt' -exec openssl x509 -text -noout -in {} \; | grep -A2 Validity

        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:29 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:52 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:06:48 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:39:56 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 10:29:43 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 19:40:13 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:36:38 2022 GMT

下一步生成新的配置文件：admin.conf、controller-manager.conf、kubelet.conf、scheduler.conf，第一步将旧文件移动到/tmp

cd /etc/kubernetes/
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/

kubeadm alpha phase kubeconfig all  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml 

[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"

重启master kubelet并重启后查看状态

sudo systemctl stop kubelet; sudo docker stop $(docker ps -aq); sudo docker rm $(docker ps -aq); sudo systemctl start kubelet

systemctl status kubelet -l

● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 14:00:22 MSK; 10s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 52998 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 53001 (kubelet)
   Memory: 51.2M
   CGroup: /system.slice/kubelet.service

检查访问主节点

kubectl get nodes

kubectl get ns

NAME                  STATUS   AGE
default               Active   464d
product1-mvp          Active   318d
infra-logging         Active   315d
infra-nginx-ingress   Active   386d
kube-public           Active   464d
kube-system           Active   464d
pg                    Active   318d

检查证书

notAfter=Mar  3 07:40:43 2022 GMT

对所有主节点重复此过程。

现在我们准备更新工作节点的证书。第一步，我们必须删除或移动 kubelet.conf

cd /etc/kubernetes/

mv kubelet.conf kubelet.conf_old

更改后bootstrap-kubelet.conf

**apiVersion: v1 集群：

集群：证书颁发机构数据：| LS0tLS1CRUDJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKYTXRRJ100 服务器：https://192.168.4.201:6443 名称：产品1 上下文：
上下文：集群：product1 用户：tls-bootstrap-token-user 名称：tls-bootstrap-token-user@product1 当前上下文：tls-bootstrap-token-user@product1 种类：配置喜好： {} 用户：
名称：tls-bootstrap-token-user 用户：代币：fgz9qz.lujw0bwsdfhdsfjhgds**

其中： certificate-authority-data – 根证书 PKI CA master，可以从 master 节点上的 /etc/kubernetes/kubelet.conf 获取

server: https://192.168.4.201:6443 - ip api server 主节点虚拟余额ip

token: fgz9qz.lujw0bwsdfhdsfjhgds - 令牌，在主节点上生成

kubeadm token create

重启并检查 kubelet 工作节点和工作节点后返回集群：

systemctl restart kubelet

systemctl status kubelet -l

● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 14:06:33 MSK; 11s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 54615 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 54621 (kubelet)
   Memory: 52.1M
   CGroup: /system.slice/kubelet.service

检查更新证书可能是目录中的 ls

ls -las /var/lib/kubelet/pki/

total 24
4 -rw-------. 1 root root 1135 Mar  3 14:06 kubelet-client-2021-03-03-14-06-34.pem
0 lrwxrwxrwx. 1 root root   59 Mar  3 14:06 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-03-03-14-06-34.pem
4 -rw-r--r--. 1 root root 2267 Mar  2 10:40 kubelet.crt
4 -rw-------. 1 root root 1679 Mar  2 10:40 kubelet.key

在所有工作节点重复此过程后。

Answer 8

我遇到了同样的问题。
在我的情况下，编码很好 - 我只是复制了没有 "=" 的最后一个字符的输出。

可能导致相同错误的其他场景：

base64 复制输出中错位的空格或折线。
未在密钥中指定 type 字段。

无法在Kubernetes中创建Secret：输入时存在非法的base64数据

8 个答案: