Question

我目前（但未成功）尝试通过istio-egressgateway设置MTL，以访问外部K8s集群服务。我正在遵循istio docs上指定的指示，但没有按预期方式工作，而且我看不到哪里出了错。

环境

VMWare ESXi下的3个VM（1个主节点，2个节点）
OS：具有来自Elrepo的内核5.2.10-1.el7的CentOS 7
k8s联网：cilium -1.6.4
证书：cert-manager-0.12 nowebhook
- ClusterIssuer CA证书=从私有PKI导入
K8s外部负载均衡器：metallb-0.8.3

在K8集群中使用的

kubectl和istio

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:37:43Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

$ istioctl version
client version: 1.2.10
citadel version: 1.2.10
egressgateway version: 1.2.10
galley version: 1.2.10
ingressgateway version: 1.2.10
pilot version: 1.2.10
policy version: 1.2.10
sidecar-injector version: 1.2.10
telemetry version: 1.2.10

istio-egressgateway 已按照istio docs

的指示进行了重新部署

应用的.yaml几乎是示例中的粘贴内容，并进行了少量修改以通过其IP地址指向外部服务

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: nginx
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  ports:
  - number: 80
    name: http
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  endpoints:
  - address: <private_ip_address>
    ports:
      https: 443
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  tls:
  - match:
    - port: 443
      sni_hosts:
      - <my.hostname>
    route:
    - destination:
        host: <my.hostname>
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: tools-istio
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - <my.hostname>
    tls:
      mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-nginx
  namespace: tools-istio
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: nginx
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: ISTIO_MUTUAL
          sni: <my.hostname>
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-nginx-through-egress-gateway
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: nginx
        port:
          number: 443
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 443
    route:
    - destination:
        host: <my.hostname>
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-nginx
  namespace: tools-istio
spec:
  host: <my.hostname>
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: ISTIO_MUTUAL
        sni: <my.hostname>
        # I've also tried with
        # mode: MUTUAL
        # clientCertificate: /etc/nginx-external-tls/tls.crt
        # privateKey: /etc/nginx-external-tls/tls.key
        # caCertificates: /etc/nginx-external-tls/ca.crt

通过curl对外部服务进行查询时，我们应该获取HTML Web代码，但是，它尝试使用HTTP而不是HTTPS执行查询，为什么？ 。调试器窗格中的istio-proxy应该使用城堡证书将HTTP转换为HTTPS

 $ kubectl -n tools-istio exec -it debugger-5844b6d674-4fvt9 -c debugger -- curl http://<my.hostname> -v
*   Trying <my.hostname>:80...
* TCP_NODELAY set
* Connected to <my.hostname> (<private_ip_address>) port 80 (#0)
> GET / HTTP/1.1
> Host: <my.hostname>
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Wed, 08 Jan 2020 09:29:09 GMT
< server: envoy
<
* Connection #0 to host <my.hostname> intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

验证proxy-status时，我得到以下信息

$ istioctl proxy-status
NAME                                                   CDS                            LDS        EDS               RDS          PILOT                            VERSION
debugger-5844b6d674-4fvt9.tools-istio                  STALE (Never Acknowledged)     SYNCED     SYNCED (51%)      SYNCED       istio-pilot-5f45f6768c-fmvr7     1.2.10
istio-egressgateway-5ff889c5fd-jtz55.istio-system      SYNCED                         SYNCED     SYNCED (100%)     SYNCED       istio-pilot-5f45f6768c-fmvr7     1.2.10
istio-ingressgateway-779cfdd879-bb7w7.istio-system     STALE (Never Acknowledged)     SYNCED     SYNCED (96%)      NOT SENT     istio-pilot-5f45f6768c-fmvr7     1.2.10

根据官方istio docs的意思可能是.yaml中的配置错误，或者是istio错误

检查端点时，一切看起来都很好

$ istioctl proxy-config endpoint debugger-5844b6d674-4fvt9.tools-istio | grep <my.hostname>
<private_ip_address>:80         HEALTHY     outbound|80||<my.hostname>
<private_ip_address>:443        HEALTHY     outbound|443||<my.hostname>

否则，调试器pod istio-proxy日志将显示以下错误。但是外部证书不是在pod级别上，而是在重新编译的istio-egressgateway中（同样，这没什么好玩的，我只是关注istio docs），那么为什么istio-proxy在寻找外部证书？？

[2020-01-08 07:56:04.827][17][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Cluster outbound|443||afv.test.recouv destroyed
[2020-01-08 07:56:04.827][17][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:73] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) outbound|443||<my.hostname>: Invalid path: /etc/nginx-external-tls/tls.crt

试验日志显示相同的istio-proxy调试器容器容器继续寻找外部服务CA证书，但是此证书位于egressgateway

中

$ kubectl -n istio-system logs istio-pilot-5f45f6768c-fmvr7 -c discovery | grep <my.hostname>

2020-01-08T08:48:54.536073Z     warn    ads     ADS:CDS: ACK ERROR 127.0.0.1:60012 sidecar~10.27.187.219~debugger-5844b6d674-4fvt9.tools-istio~tools-istio.svc.cluster.local-15433 (debugger-5844b6d674-4fvt9.tools-istio) version_info:"2020-01-08T07:50:00Z/110" node:<id:"sidecar~10.27.187.219~debugger-5844b6d674-4fvt9.tools-istio~tools-istio.svc.cluster.local" cluster:"debugger.tools-istio" metadata:<fields:<key:"CONFIG_NAMESPACE" value:<string_value:"tools-istio" > > fields:<key:"INCLUDE_INBOUND_PORTS" value:<string_value:"8080" > > fields:<key:"INTERCEPTION_MODE" value:<string_value:"REDIRECT" > > fields:<key:"ISTIO_META_INSTANCE_IPS" value:<string_value:"10.27.187.219,10.27.187.219,fe80::a07f:d0ff:feec:5e71" > > fields:<key:"ISTIO_PROXY_SHA" value:<string_value:"istio-proxy:f9c0feb10cc42277e97dd080a2e045f62d1739a9" > > fields:<key:"ISTIO_PROXY_VERSION" value:<string_value:"1.1.3" > > fields:<key:"ISTIO_VERSION" value:<string_value:"1.2.10" > > fields:<key:"POD_NAME" value:<string_value:"debugger-5844b6d674-4fvt9" > > fields:<key:"app" value:<string_value:"debugger" > > fields:<key:"istio" value:<string_value:"sidecar" > > fields:<key:"pod-template-hash" value:<string_value:"5844b6d674" > > fields:<key:"prometheus.io/path" value:<string_value:"/metrics" > > fields:<key:"prometheus.io/port" value:<string_value:"8080" > > fields:<key:"prometheus.io/scrape" value:<string_value:"true" > > fields:<key:"sidecar.istio.io/rewriteAppHTTPProbers" value:<string_value:"true" > > > locality:<> build_version:"f9c0feb10cc42277e97dd080a2e045f62d1739a9/1.11.0-dev/Modified/RELEASE/BoringSSL" > type_url:"type.googleapis.com/envoy.api.v2.Cluster" response_nonce:"59a1bb9a-aca6-47c0-b718-a86d7e392db0" error_detail:<code:13 message:"Error adding/updating cluster(s) outbound|443||afv.test.recouv: Invalid path: /etc/nginx-external-tls/tls.crt" >
  message: "Error adding/updating cluster(s) outbound|443||<my.hostname>: Invalid path: /etc/nginx-external-tls/tls.crt"

在istio-egressgateway中，一切都正常了

$ kubectl -n istio-system logs istio-egressgateway-5ff889c5fd-jtz55
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:527] updating TLS cluster outbound_.80_._.<my.hostname>
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:975] membership update for TLS cluster outbound_.80_._.<my.hostname> added 1 removed 0
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:527] updating TLS cluster outbound_.443_._.<my.hostname>
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:975] membership update for TLS cluster outbound_.443_._.<my.hostname> added 1 removed 0

当然，证书也应在istio-egressgateway中出现。我曾经使用过一个证书管理器集群颁发者来为外部服务提供证书，就像AC及其机密已经在集群中一样

# External AC
$ kubectl -n istio-system exec -it istio-egressgateway-5ff889c5fd-jtz55 -- ls /etc/cluster-issuer-tls
tls.crt  tls.key

# Client
$ kubectl -n istio-system exec -it istio-egressgateway-5ff889c5fd-jtz55 -- ls /etc/nginx-external-tls
ca.crt  tls.crt  tls.key

有人对我所缺少的东西有任何线索吗？这是配置错误还是错误？

任何帮助将不胜感激，我从圣诞节假期开始就一直坚持这样做

欢呼声

用于外部服务的Istio Egress MTL

0 个答案: