Kubernetes:Pods无法解析主机名

时间:2017-08-21 21:04:12

标签: dns kubernetes

我遇到了Kubernetes的问题,我的pod无法解析主机名(例如google.comkubernetes.default)。

我目前在OpenStack中的两个CentOS7实例上运行了1个主节点和1个节点。我使用kubeadm进行了部署。

以下是安装的版本:

kubeadm-1.7.3-1.x86_64
kubectl-1.7.3-1.x86_64
kubelet-1.7.3-1.x86_64
kubernetes-cni-0.5.1-0.x86_64

下面概述了一些验证步骤,可能会对我的问题有所了解。

我定义了一个busybox pod:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

然后创建pod:

$ kubectl create -f busybox.yaml

尝试执行名称为google.com的DNS查找:

$ kubectl exec -ti busybox -- nslookup google.com
Server:    10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'google.com'

尝试执行名称为kubernetes.default的DNS查找:

$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes.default'

检查我的DNS pod是否正在运行:

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                        READY     STATUS    RESTARTS   AGE
kube-dns-2425271678-k1nft   3/3       Running   9          5d

检查我的DNS服务是否已启动:

$ kubectl get svc --namespace=kube-system
NAME       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   10.96.0.10   <none>        53/UDP,53/TCP   5d

检查DNS端点是否暴露:

$ kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                     AGE
kube-dns   10.244.0.5:53,10.244.0.5:53   5d

检查我容器中/etc/resolv.conf的内容:

$ kubectl exec -ti busybox -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

如果我理解正确,Kubernetes文档说明我的pod应该继承节点(或master?)的DNS配置。但是,即使只有一行(nameserver 10.92.128.40),我在旋转播放时会收到以下警告:

Search Line limits were exceeded, some dns names have been omitted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local mydomain.net anotherdomain.net yetanotherdomain.net

我知道存在一个已知问题,/etc/resolv.conf中只能列出这么多项目。但是,我的容器中的上述搜索行和名称服务器将从哪里生成?

最后,这是来自kube-dns容器的日志:

$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
I0817 20:54:58.445280       1 dns.go:48] version: 1.14.3-4-gee838f6
I0817 20:54:58.452551       1 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
I0817 20:54:58.452616       1 server.go:113] FLAG: --alsologtostderr="false"
I0817 20:54:58.452628       1 server.go:113] FLAG: --config-dir="/kube-dns-config"
I0817 20:54:58.452638       1 server.go:113] FLAG: --config-map=""
I0817 20:54:58.452643       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0817 20:54:58.452650       1 server.go:113] FLAG: --config-period="10s"
I0817 20:54:58.452659       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0817 20:54:58.452665       1 server.go:113] FLAG: --dns-port="10053"
I0817 20:54:58.452674       1 server.go:113] FLAG: --domain="cluster.local."
I0817 20:54:58.452683       1 server.go:113] FLAG: --federations=""
I0817 20:54:58.452692       1 server.go:113] FLAG: --healthz-port="8081"
I0817 20:54:58.452698       1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I0817 20:54:58.452704       1 server.go:113] FLAG: --kube-master-url=""
I0817 20:54:58.452713       1 server.go:113] FLAG: --kubecfg-file=""
I0817 20:54:58.452718       1 server.go:113] FLAG: --log-backtrace-at=":0"
I0817 20:54:58.452727       1 server.go:113] FLAG: --log-dir=""
I0817 20:54:58.452734       1 server.go:113] FLAG: --log-flush-frequency="5s"
I0817 20:54:58.452741       1 server.go:113] FLAG: --logtostderr="true"
I0817 20:54:58.452746       1 server.go:113] FLAG: --nameservers=""
I0817 20:54:58.452752       1 server.go:113] FLAG: --stderrthreshold="2"
I0817 20:54:58.452759       1 server.go:113] FLAG: --v="2"
I0817 20:54:58.452765       1 server.go:113] FLAG: --version="false"
I0817 20:54:58.452775       1 server.go:113] FLAG: --vmodule=""
I0817 20:54:58.452856       1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I0817 20:54:58.453680       1 server.go:198] Skydns metrics enabled (/metrics:10055)
I0817 20:54:58.453692       1 dns.go:147] Starting endpointsController
I0817 20:54:58.453699       1 dns.go:150] Starting serviceController
I0817 20:54:58.453841       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0817 20:54:58.453852       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0817 20:54:58.964468       1 dns.go:171] Initialized services and endpoints from apiserver
I0817 20:54:58.964523       1 server.go:129] Setting up Healthz Handler (/readiness)
I0817 20:54:58.964536       1 server.go:134] Setting up cache handler (/cache)
I0817 20:54:58.964545       1 server.go:120] Status HTTP port 8081

dnsmasq容器。无视它发现了几个名字服务器而不仅仅是我所说的resolv.conf中的名字服务器,因为我原来的确有更多。我试图通过删除附加内容来简单地使用它:

$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
I0817 20:55:03.295826       1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0817 20:55:03.298134       1 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0817 20:55:03.731577       1 nanny.go:111] 
W0817 20:55:03.731609       1 nanny.go:112] Got EOF from stdout
I0817 20:55:03.731642       1 nanny.go:108] dnsmasq[9]: started, version 2.76 cachesize 1000
I0817 20:55:03.731656       1 nanny.go:108] dnsmasq[9]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0817 20:55:03.731681       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
I0817 20:55:03.731689       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
I0817 20:55:03.731695       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain cluster.local 
I0817 20:55:03.731704       1 nanny.go:108] dnsmasq[9]: reading /etc/resolv.conf
I0817 20:55:03.731710       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
I0817 20:55:03.731717       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
I0817 20:55:03.731723       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain cluster.local 
I0817 20:55:03.731729       1 nanny.go:108] dnsmasq[9]: using nameserver 10.92.128.40#53
I0817 20:55:03.731735       1 nanny.go:108] dnsmasq[9]: using nameserver 10.92.128.41#53
I0817 20:55:03.731741       1 nanny.go:108] dnsmasq[9]: using nameserver 10.95.207.66#53
I0817 20:55:03.731747       1 nanny.go:108] dnsmasq[9]: read /etc/hosts - 7 addresses

sidecar容器:

$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
ERROR: logging before flag.Parse: I0817 20:55:04.488391       1 main.go:48] Version v1.14.3-4-gee838f6
ERROR: logging before flag.Parse: I0817 20:55:04.488612       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
ERROR: logging before flag.Parse: I0817 20:55:04.488667       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: I0817 20:55:04.488766       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}

我大部分时间都在阅读group join提供的文件。任何方向,见解或尝试的事情都将非常感激。

4 个答案:

答案 0 :(得分:5)

我有类似的问题。重新启动coredns部署为我解决了这个问题:

kubectl -n kube-system rollout restart deployment coredns

答案 1 :(得分:3)

有些想法浮现在脑海中:

答案 2 :(得分:3)

遇到同样的问题。我遵循了该文档dns-debugging-resolution,并检查了与DNS相关的Pod,服务,端点,它们都在运行,并且没有错误消息。最终,我发现我的 calico 服务已终止。在我启动印花布服务并等待了几分钟后,它开始工作。

答案 3 :(得分:1)

如果发现类似以下错误,请检查coredns pods日志:

# kubectl logs --namespace=kube-system coredns-XXX
  ...
  [ERROR] plugin/errors ... HINFO: read udp ... read: no route to host

然后确保主机上启用了防火墙化装:

# firewall-cmd --list-all
  ... 
  masquerade: yes

Enable if it's "no":
# firewall-cmd --add-masquerade --permanent
# firewall-cmd --reload

*此后您可能需要重新启动/重启