Question

我刚刚将Ubuntu从19.04升级到19.10

现在Minikube无法启动。

所以，过了一会儿，我才将Minikube完全移除了。

minikube stop; minikube delete
docker stop $(docker ps -aq)
rm -r ~/.kube ~/.minikube
sudo rm /usr/local/bin/localkube /usr/local/bin/minikube
systemctl stop '*kubelet*.mount'
sudo rm -rf /etc/kubernetes/
docker system prune -af --volumes

现在，我想重新安装所有内容，但无法使其正常运行。

我下载了minikube，并将其移至/usr/local/bin

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \\n  && chmod +x minikube && sudo mv ./minikube /usr/local/bin

我开始玩minikube

sudo minikube start --vm-driver=none

一切正常，minukube成功启动。

~ sudo minikube start --vm-driver=none
?  minikube v1.4.0 on Ubuntu 19.10
?  Running on localhost (CPUs=4, Memory=7847MB, Disk=280664MB) ...
ℹ️   OS release is Ubuntu 19.10
?  Preparing Kubernetes v1.16.0 on Docker 18.09.6 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
?  Pulling images ...
?  Launching Kubernetes ... 
?  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
?  https://minikube.sigs.k8s.io/docs/reference/drivers/none/

⚠️  kubectl and minikube configuration will be stored in /root
⚠️  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

?  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
⌛  Waiting for: apiserver proxy etcd scheduler controller dns
?  Done! kubectl is now configured to use "minikube"

我终于做到了：

   ~ sudo mv /root/.kube /root/.minikube $HOME
➜  ~ sudo chown -R $USER $HOME/.kube $HOME/.minikube

但是当我想检查豆荚时：

kubectl get po

我得到：

➜  ~ kubectl get po
Error in configuration: 
* unable to read client-cert /root/.minikube/client.crt for minikube due to open /root/.minikube/client.crt: permission denied
* unable to read client-key /root/.minikube/client.key for minikube due to open /root/.minikube/client.key: permission denied
* unable to read certificate-authority /root/.minikube/ca.crt for minikube due to open /root/.minikube/ca.crt: permission denied

如果使用sudo：

 ~ sudo kubectl get po
[sudo] password for julien: 
The connection to the server localhost:8080 was refused - did you specify the right host or port?

这是minikube logs

的结果

https://gist.github.com/xoco70/8a9c7042238400e370796cb23cb11c88

我该怎么办？

编辑：

重启后，使用以下命令启动minikube时：

sudo minikube start --vm-driver=none

我得到：

Error starting cluster: cmd failed: sudo env PATH=/var/lib/minikube/binaries/v1.16.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap

: running command: sudo env PATH=/var/lib/minikube/binaries/v1.16.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
 output: [init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
    [WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
    [WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
    [WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
    [WARNING FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [WARNING Swap]: running with swap on is not supported. Please disable swap
    [WARNING FileExisting-ethtool]: ethtool not found in system path
    [WARNING FileExisting-socat]: socat not found in system path
    [WARNING Hostname]: hostname "minikube" could not be reached
    [WARNING Hostname]: hostname "minikube": lookup minikube on 8.8.8.8:53: no such host
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
    [WARNING Port-10250]: Port 10250 is in use
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR Port-8443]: Port 8443 is in use
    [ERROR Port-10251]: Port 10251 is in use
    [ERROR Port-10252]: Port 10252 is in use
    [ERROR Port-2379]: Port 2379 is in use
    [ERROR Port-2380]: Port 2380 is in use
    [ERROR DirAvailable--var-lib-minikube-etcd]: /var/lib/minikube/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
: running command: sudo env PATH=/var/lib/minikube/binaries/v1.16.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
.: exit status 1

?  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
?  https://github.com/kubernetes/minikube/issues/new/choose
❌  Problems detected in kube-apiserver [3e0d8c59345d]:
    I1025 07:09:56.349120       1 log.go:172] http: TLS handshake error from 127.0.0.1:46254: remote error: tls: bad certificate
    I1025 07:09:56.353714       1 log.go:172] http: TLS handshake error from 127.0.0.1:46082: remote error: tls: bad certificate
    I1025 07:09:56.353790       1 log.go:172] http: TLS handshake error from 127.0.0.1:46080: remote error: tls: bad certificate

Answer 1

一个令人困惑的问题，就是您的故事中有一些不匹配的地方。

您正在询问如何“修复Minikube并保留我的群集”，同时提到您删除了本地群集，但是仍然存在问题。

我只能在这里假设您终于打破了群集，无需恢复它。

看，您说尽管您的集群尚未启动（Error restarting cluster:exit status 1，Sorry that minikube crashed.错误），但命令sudo kubectl get po -A仍然有效。而且在普通用户下执行相同操作会遇到一些问题。

这里的问题-“ sudo kubectl get po -A仍然有效”是什么意思？你看到豆荚了吗？你还能看到其他物体吗？

如果是这样，您首先应该研究etcd direction。此外，您在启动过程中遇到了etcd个错误。

: running command: sudo ln -s /data/minikube /var/lib/minikube/etcd
 output: ln: failed to create symbolic link '/var/lib/minikube/etcd/minikube': File exists
: running command: sudo ln -s /data/minikube /var/lib/minikube/etcd
.: exit status 1

Etcd是k8的主键值数据存储，用于存储和复制所有kubernetes集群状态，很可能因此我认为存在问题。

也很想知道您删除集群的方式。我敢打赌，您已经完成minikube delete并忘记了手动删除不必要的配置。正确的方法：

minikube stop
minikube delete
rm -rf ~/.kube
rm -rf ~/.minikube
sudo rm -rf /var/lib/minikube
sudo rm /var/lib/kubeadm.yaml
sudo rm -rf /etc/kubernetes

删除所有内容并从头开始安装。

Answer 2

好吧，所以我将minikube复制到19.10后进行了复制并出现了相同的错误。

我如何在19.04上启动集群：

#Install kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl

#Install minikube. Make sure to check for latest version
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

#Install Docker
curl -fsSL get.docker.com -o get-docker.sh && chmod +x get-docker.sh
sh get-docker.sh
sudo usermod -aG docker $USER


export MINIKUBE_WANTUPDATENOTIFICATION=false
export MINIKUBE_WANTREPORTERRORPROMPT=false
export MINIKUBE_HOME=$HOME
export CHANGE_MINIKUBE_NONE_USER=true
export KUBECONFIG=$HOME/.kube/config
sudo minikube start --vm-driver none
sudo chown -R $USER $HOME/.kube $HOME/.minikube


vkr@ubuntu-minikube:~$ docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2f86
 Built:             Tue Oct  8 01:00:44 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2f86
  Built:            Tue Oct  8 00:59:17 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683


vkr@ubuntu-minikube:~$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-5644d7b6d9-cv8c5           1/1     Running   0          2m25s
kube-system   coredns-5644d7b6d9-gk725           1/1     Running   0          2m25s
kube-system   etcd-minikube                      1/1     Running   0          75s
kube-system   kube-addon-manager-minikube        1/1     Running   0          75s
kube-system   kube-apiserver-minikube            1/1     Running   0          98s
kube-system   kube-controller-manager-minikube   1/1     Running   0          88s
kube-system   kube-proxy-59jp9                   1/1     Running   0          2m25s
kube-system   kube-scheduler-minikube            1/1     Running   0          82s
kube-system   storage-provisioner                1/1     Running   0          2m24s

升级到19.10并清洁minikube后，安装：

vkr@ubuntu-minikube:~$ kubectl get all -A
Error in configuration: 
* unable to read client-cert /root/.minikube/client.crt for minikube due to open /root/.minikube/client.crt: permission denied
* unable to read client-key /root/.minikube/client.key for minikube due to open /root/.minikube/client.key: permission denied
* unable to read certificate-authority /root/.minikube/ca.crt for minikube due to open /root/.minikube/ca.crt: permission denied

有很多讨论表明您应该对root使用none driver，因为minikube直接在您的机器上运行kubernetes系统组件...

Running minikube as normal user

Can't start minikube-- permissions

https://minikube.sigs.k8s.io/docs/reference/drivers/none/：

用法none驱动程序要求minikube以root身份运行，直到 #3760可以是已解决

但是..这是一个小技巧。

1）擦拭一切

vkr@ubuntu-minikube:~$ minikube stop
✋  Stopping "minikube" in none ...
?  "minikube" stopped.
vkr@ubuntu-minikube:~$ minikube delete
?  Uninstalling Kubernetes v1.16.0 using kubeadm ...
?  Deleting "minikube" in none ...
?  The "minikube" cluster has been deleted.
vkr@ubuntu-minikube:~$ rm -rf ~/.kube
vkr@ubuntu-minikube:~$ rm -rf ~/.minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /var/lib/minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /etc/kubernetes
vkr@ubuntu-minikube:~$ sudo rm -rf /root/.minikube
vkr@ubuntu-minikube:~$ sudo rm -rf /usr/local/bin/minikube

2）安装minikube，导出变量，检查

vkr@ubuntu-minikube:~$  curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
vkr@ubuntu-minikube:~$   export MINIKUBE_WANTUPDATENOTIFICATION=false
vkr@ubuntu-minikube:~$  export MINIKUBE_WANTREPORTERRORPROMPT=false
vkr@ubuntu-minikube:~$  export MINIKUBE_HOME=$HOME
vkr@ubuntu-minikube:~$  export CHANGE_MINIKUBE_NONE_USER=true
vkr@ubuntu-minikube:~$  export KUBECONFIG=$HOME/.kube/config
vkr@ubuntu-minikube:~$  sudo minikube start --vm-driver none

?  minikube v1.4.0 on Ubuntu 19.10
?  Running on localhost (CPUs=2, Memory=7458MB, Disk=9749MB) ...
ℹ️   OS release is Ubuntu 19.10
?  Preparing Kubernetes v1.16.0 on Docker 19.03.3 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
?  Downloading kubelet v1.16.0
?  Downloading kubeadm v1.16.0
?  Pulling images ...
?  Launching Kubernetes ... 
?  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
?  https://minikube.sigs.k8s.io/docs/reference/drivers/none/

⚠️  kubectl and minikube configuration will be stored in /root
⚠️  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

?  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
⌛  Waiting for: apiserver proxy etcd scheduler controller dns
?  Done! kubectl is now configured to use "minikube"

接下来我要做的是将所有内容从/root/.kube和/root/.minikube复制到$HOME，授予用户权限，最后编辑$HOME/.kube/config，指定证书的新路径（{{1} }代替$HOME/.minikube/）。现在看起来像

/root/.minikube/

让我们这样做：）

vkr@ubuntu-minikube:~$ cat $KUBECONFIG
apiVersion: v1
...
    certificate-authority: /root/.minikube/ca.crt
...
    client-certificate: /root/.minikube/client.crt
    client-key: /root/.minikube/client.key

最后就是结果。

vkr@ubuntu-minikube:~$ sudo cp -r /root/.kube  /root/.minikube $HOME
vkr@ubuntu-minikube:~$ sudo chown -R $USER $HOME/.kube
vkr@ubuntu-minikube:~$ sudo chown -R $USER $HOME/.minikube
sed 's/root/home\/vkr/g' $KUBECONFIG > tmp; mv tmp $KUBECONFIG

在Ubuntu升级到19.10之后，Minikube将无法运行

2 个答案: