CoreDNS无法使用K8S / Minikube解析名称空间外部的服务URL

时间:2020-03-19 10:37:23

标签: kubernetes dns

我有一个运行minikube 1.6.2的本地集群。

我所有的pod都还可以,我分别检查了日志,但是我有2个db,influx和postgres,不能再从名称空间之外的任何URL访问。

我登录了两个Pod,可以确认每个数据库都正常,有数据,并且可以与用户/ pass手动连接。

以涌入为例。

kubectl exec -it -n influx blockchain-influxdb-local-fb745b98c-vbghp -- influx -username='myuser' -password="mypass" -database="mydb" -precision=rfc3339 -execute "show measurements"

给我4次测量,所以没有pb。

但是当我尝试将来自同一个名称空间的Influx与他的本地dns连接时,我会超时。

➜ ~ kubectl get svc -n influx 
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
blockchain-influxdb-local   ClusterIP   10.96.175.62   <none>        8086/TCP    19m


➜ ~ kubectl get deployments -n influx
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
blockchain-influxdb-local   1/1     1            1           20m


➜  ~ kubectl get po -n influx
NAME                                        READY   STATUS    RESTARTS   AGE
blockchain-influxdb-local-fb745b98c-vbghp   1/1     Running   0          21m
measures-api-local-8667bb496f-4wp8d         1/1     Running   0          21m

适用的情况:

从同一个命名空间中的pod:

curl --verbose -G 'http://blockchain-influxdb-local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

从另一个命名空间(相同命名空间)中的Pod,使用Pod IP

curl --verbose -G '172.17.0.5:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

从另一个名称空间(相同名称空间)中的Pod,通过服务IP

curl --verbose -G '10.96.175.62:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

但是当我从外部名称空间中使用本地dns时,它将无法正常工作,我会从CURL中收到超时:

 curl --verbose -G 'blockchain-influxdb-local.influx.svc.cluster.local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

我遵循了这些调试步骤,以确保DNS正常运行,并且没有问题,一切正常。

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

在同一个容器中,当我ping通此URL时,我得到:

root@metadata-api-local-8b4b7846b-zllb8:/go/src/gitlab.com/company/metadata_api# ping blockchain-influxdb-local.influx.svc.cluster.local

PING nc-ass-vip.sdv.fr (212.95.74.75) 56(84) bytes of data.
--- nc-ass-vip.sdv.fr ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 47ms

我不知道为什么要引用nc-ass-vip.sdv.fr

我还试图删除本地群集并重新部署它,还尝试将minikube更新到最新版本(1.8.2),无济于事。

我不知道该怎么办...

有人有想法吗?我工作了好几个月,真的不知道发生了什么。 :(

作为对@Arghya Sadhu的回应,我从Influx窗格中发布了/etc/resolv.conf文件:

nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:5

kubectl edit cm coredns -n kube-system

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-03-19T10:59:28Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "176"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 0797c1a9-e9db-4b4c-bc8d-4c7ecca24968

编辑:

kubectl exec -ti dnsutils -- nslookup blockchain-influxdb-local.influx.svc.cluster.local                                         
Server:     10.96.0.10
Address:    10.96.0.10#53

Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr   canonical name = nc-ass-vip.sdv.fr.
Name:   nc-ass-vip.sdv.fr
Address: 212.95.74.75

1 个答案:

答案 0 :(得分:1)

在研究了几种可能性之后,我们发现了以下命令的输出:

$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.

这些命令here的输出(添加到该答案的末尾,以防将来链接时无法参考)。

通过查看此输出,我们可以看出,无论numericable.fr为何总是对dns查询给出肯定的答案。

为避免这种情况,您可以将窗格中的ndots条目更改为1甚至0。

nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:0

在手册页中,我们有:

ndots:n 设置必须出现的点数的阈值 在给res_query(3)的名称中(请参阅resolver(3)) 将进行初始绝对查询。的默认 n为1,表示如果名称中有点, 名称将首先作为绝对名称尝试 搜索列表元素将附加到该元素。的价值 此选项默认为15。

一个更有效和长期的解决方案是将此条目添加到pod / statefulset / deployment清单中,如以下示例所示:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsConfig:
    options:
      - name: ndots
        value: "0"

参考的命令输出,以供将来参考:

➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx 

Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 10
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.default.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
    ANSWERS:
    ->  blockchain-influxdb-local.influx.svc.cluster.local
    internet address = 10.96.72.6
    ttl = 10
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6

pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local 

Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr, type = A, class = IN
    ANSWERS:
    ->  blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr
    canonical name = nc-ass-vip.sdv.fr.
    ttl = 30
    ->  nc-ass-vip.sdv.fr
    internet address = 212.95.74.75
    ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr   canonical name = nc-ass-vip.sdv.fr.
Name:   nc-ass-vip.sdv.fr
Address: 212.95.74.75

pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
    ANSWERS:
    ->  blockchain-influxdb-local.influx.svc.cluster.local
    internet address = 10.96.72.6
    ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6

pod "dnsutils" deleted