Question

我正在尝试通过ingress-nginx和Google云负载均衡器公开GKE集群中的mlflow模型。

各个部署的服务配置如下：

apiVersion: v1
kind: Service
metadata:
  name: model-inference-service
  labels:
    app: inference
spec:
  ports:
  - port: 5555
    targetPort: 5555
  selector:
    app: inference

使用kubectl port-forward service/model-inference-service 5555:5555将此服务转发到localhost时，可以使用以下script将测试图像发送到api端点，从而成功查询模型。

请求发送到的URL为http://127.0.0.1:5555/invocations。 这按预期工作，因此我假设运行运行Pod的部署公开了模型，并且相应的clusterIP服务model-inference-service已正确配置。

接下来，我通过这样做将ingress-nxinx安装到集群中

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install my-release ingress-nginx/ingress-nginx

入口的配置如下（我怀疑错误一定在这里吗？）：

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
#    nginx.ingress.kubernetes.io/rewrite-target: /invocations
  name: inference-ingress
  namespace: default
  labels:
    app: inference
spec:
  rules:
    - http:
        paths:
          - path: /invocations
            backend:
              serviceName: model-inference-service
              servicePort: 5555

入口控制器pod运行成功：

my-release-ingress-nginx-controller-6758cc8f45-fwtw7   1/1     Running   0          3h33m

在GCP控制台中，我可以看到负载均衡器也已成功创建，并且可以获取其IP。

当使用我之前向Rest api端点发出请求（以前该服务已转发到localhost）但现在使用负载均衡器的ip的测试脚本时，出现502 Bad Gateway错误：

现在的网址如下：http://34.90.4.0:80/invocations

Traceback (most recent call last):
  File "test_inference.py", line 80, in <module>
    run()
  File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "test_inference.py", line 76, in run
    print(score_model(data_path, host, port).text)
  File "test_inference.py", line 54, in score_model
    status_code=response.status_code, text=response.text
Exception: Status Code 502. <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>

在浏览器中访问相同的URL时，它说：

502 Bad Gateway
nginx/1.19.1

入口控制器状态的日志：

2020/08/26 16:06:45 [warn] 86#86: *42282 a client request body is buffered to a temporary file /tmp/client-body/0000000009, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", host: "34.90.4.0"
2020/08/26 16:06:45 [error] 86#86: *42282 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", upstream: "http://10.52.3.7:5555/invocations", host: "34.90.4.0"
2020/08/26 16:06:45 [error] 86#86: *42282 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", upstream: "http://10.52.3.7:5555/invocations", host: "34.90.4.0"
2020/08/26 16:06:45 [error] 86#86: *42282 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", upstream: "http://10.52.3.7:5555/invocations", host: "34.90.4.0"
10.10.0.30 - - [26/Aug/2020:16:06:45 +0000] "POST /invocations HTTP/1.1" 502 157 "-" "python-requests/2.24.0" 86151 0.738 [default-model-inference-service-5555] [] 10.52.3.7:5555, 10.52.3.7:5555, 10.52.3.7:5555 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 0d86e360427c0a81c287da4ff5e907bc

要测试入口和负载均衡器是否在原理上工作，我用想要公开的真实REST API替换了docker映像，该docker image会在端口5050和路径{{1上返回“ hello world” }}。我在上面显示的服务和入口清单中更改了端口和路径（从/到/invocations），并且在浏览器中访问负载平衡器的ip时可以成功看到“ hello world”。 / p>

有人看到我做错了吗？非常感谢！

最诚挚的问候，

F

Answer 1

您共享的配置看起来不错。群集环境中一定有某种原因导致此现象。查看Pod到Pod的通讯是否正常。在与Nginx入口控制器相同的节点上启动测试容器，并执行从该容器到目标服务的curl。查看是否遇到任何DNS或网络问题。调用服务时尝试更改主机标头，看看它是否对此敏感。

GKE配置502错误网关上的Ingress-nginx

1 个答案: