Google容器引擎（Kubernetes）：Websocket（Socket.io）无法处理多个副本

时间：2016-11-22 13:43:01

标签： node.js nginx socket.io kubernetes google-kubernetes-engine

我是Google容器引擎（GKE）的新手。在$out上运行时，它运行正常，但是当我使用GKE部署到生产时，我遇到了websocket错误。

我的节点应用程序是使用localhost和Hapi.js开发的，我的结构如下图所示。

Application Architecture

我使用Glue编写Hapi服务器。以下是我的Socket.io 

manifest.json

我的{ ... "connections": [ { "host": "app", "address": "0.0.0.0", "port": 8000, "labels": ["api"], "routes": { "cors": false, "security": { "hsts": false, "xframe": true, "xss": true, "noOpen": true, "noSniff": true } }, "router": { "stripTrailingSlash": true }, "load": { "maxHeapUsedBytes": 1073741824, "maxRssBytes": 1610612736, "maxEventLoopDelay": 5000 } }, { "host": "app", "address": "0.0.0.0", "port": 8099, "labels": ["web"], "routes": { "cors": true, "security": { "hsts": false, "xframe": true, "xss": true, "noOpen": true, "noSniff": true } }, "router": { "stripTrailingSlash": true }, "load": { "maxHeapUsedBytes": 1073741824, "maxRssBytes": 1610612736, "maxEventLoopDelay": 5000 } }, { "host": "app", "address": "0.0.0.0", "port": 8999, "labels": ["admin"], "routes": { "cors": true, "security": { "hsts": false, "xframe": true, "xss": true, "noOpen": true, "noSniff": true } }, "router": { "stripTrailingSlash": true }, "load": { "maxHeapUsedBytes": 1073741824, "maxRssBytes": 1610612736, "maxEventLoopDelay": 5000 }, "state": { "ttl": null, "isSecure": false, "isHttpOnly": true, "path": null, "domain": null, "encoding": "none", "clearInvalid": false, "strictHeader": true } } ], ... } 

nginx.conf

Kubernetes服务worker_processes 5; ## Default: 1 worker_rlimit_nofile 8192; error_log /dev/stdout info; events { worker_connections 4096; ## Default: 1024 } http { access_log /dev/stdout; server { listen 80 default_server; listen [::]:80 default_server; # Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response. return 301 https://$host$request_uri; } server { listen 443 ssl default_server; listen [::]:443 ssl default_server; server_name _; # Configure ssl ssl_certificate /etc/secret/ssl/myapp.com.csr; ssl_certificate_key /etc/secret/ssl/myapp.com.key; include /etc/nginx/ssl-params.conf; } server { listen 443 ssl; listen [::]:443 ssl; server_name api.myapp.com; location / { proxy_pass http://api_app/; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # Handle Web Socket connections proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } } server { listen 443 ssl; listen [::]:443 ssl; server_name myapp.com; location / { proxy_pass http://web_app/; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # Handle Web Socket connections proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } } server { listen 443 ssl; listen [::]:443 ssl; server_name admin.myapp.com; location / { proxy_pass http://admin_app/; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # Handle Web Socket connections proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } } # Define your "upstream" servers - the # servers request will be sent to upstream api_app { server localhost:8000; } upstream web_app { server localhost:8099; } upstream admin_app { server localhost:8999; } } 

app-service.yaml

Kubernetes部署apiVersion: v1 kind: Service metadata: name: app-nginx labels: app: app-nginx spec: type: LoadBalancer ports: # The port that this service should serve on. - port: 80 targetPort: 80 protocol: TCP name: http - port: 443 targetPort: 443 protocol: TCP name: https # Label keys and values that must match in order to receive traffic for this service. selector: app: app-nginx 

app-deployment.yaml

我正在使用apiVersion: extensions/v1beta1 kind: Deployment metadata: name: app-nginx spec: replicas: 3 template: metadata: labels: app: app-nginx spec: containers: - name: nginx image: us.gcr.io/myproject/nginx ports: - containerPort: 80 name: http - containerPort: 443 name: https volumeMounts: # This name must match the volumes.name below. - name: ssl-secret readOnly: true mountPath: /etc/secret/ssl - name: app image: us.gcr.io/myproject/bts-server ports: - containerPort: 8000 name: api - containerPort: 8099 name: web - containerPort: 8999 name: admin volumeMounts: # This name must match the volumes.name below. - name: client-secret readOnly: true mountPath: /etc/secret/client - name: admin-secret readOnly: true mountPath: /etc/secret/admin volumes: - name: ssl-secret secret: secretName: ssl-key-secret - name: client-secret secret: secretName: client-key-secret - name: admin-secret secret: secretName: admin-key-secret 。

从浏览器控制台获取错误：

Cloudflare SSL full strict

这是Nginx的日志：

WebSocket connection to 'wss://api.myapp.com/socket.io/?EIO=3&transport=websocket&sid=4Ky-y9K7J0XotrBFAAAQ' failed: WebSocket is closed before the connection is established.
https://api.myapp.com/socket.io/?EIO=3&transport=polling&t=LYByND2&sid=4Ky-y9K7J0XotrBFAAAQ Failed to load resource: the server responded with a status of 400 ()
VM50:35 WebSocket connection to 'wss://api.myapp.com/socket.io/?EIO=3&transport=websocket&sid=FsCGx-UE7ohrsSSqAAAT' failed: Error during WebSocket handshake: Unexpected response code: 502WrappedWebSocket @ VM50:35WS.doOpen @ socket.io.js:6605Transport.open @ socket.io.js:4695Socket.probe @ socket.io.js:3465Socket.onOpen @ socket.io.js:3486Socket.onHandshake @ socket.io.js:3546Socket.onPacket @ socket.io.js:3508(anonymous function) @ socket.io.js:3341Emitter.emit @ socket.io.js:6102Transport.onPacket @ socket.io.js:4760callback @ socket.io.js:4510(anonymous function) @ socket.io.js:5385exports.decodePayloadAsBinary @ socket.io.js:5384exports.decodePayload @ socket.io.js:5152Polling.onData @ socket.io.js:4514(anonymous function) @ socket.io.js:4070Emitter.emit @ socket.io.js:6102Request.onData @ socket.io.js:4231Request.onLoad @ socket.io.js:4312xhr.onreadystatechange @ socket.io.js:4184
socket.io.js:4196 GET https://api.myapp.com/socket.io/?EIO=3&transport=polling&t=LYByNpy&sid=FsCGx-UE7ohrsSSqAAAT 400 ()

更新

我在[22/Nov/2016:12:10:19 +0000] "GET /socket.io/?EIO=3&transport=websocket&sid=MGc--oncQbQI6NOZAAAX HTTP/1.1" 101 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" 10.8.0.1 - - [22/Nov/2016:12:10:19 +0000] "POST /socket.io/?EIO=3&transport=polling&t=LYByQBw&sid=MGc--oncQbQI6NOZAAAX HTTP/1.1" 200 2 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" 10.128.0.2 - - [22/Nov/2016:12:10:20 +0000] "GET /socket.io/?EIO=3&transport=polling&t=LYByQKp HTTP/1.1" 200 101 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" 10.8.0.1 - - [22/Nov/2016:12:10:21 +0000] "GET /socket.io/?EIO=3&transport=polling&t=LYByQWo&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1" 200 24 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" 2016/11/22 12:10:21 [error] 6#6: *157 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.0.1, server: api.myapp.com, request: "GET /socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1", upstream: "http://[::1]:8000/socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY", host: "api.myapp.com", referrer: "https://myapp.com/" 2016/11/22 12:10:21 [warn] 6#6: *157 upstream server temporarily disabled while connecting to upstream, client: 10.8.0.1, server: api.myapp.com, request: "GET /socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1", upstream: "http://[::1]:8000/socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY", host: "api.myapp.com", referrer: "https://myapp.com/" 10.8.0.1 - - [22/Nov/2016:12:10:22 +0000] "GET /socket.io/?EIO=3&transport=polling&t=LYByQaN&sid=c5nkusT9fEPRsu2rAAAY HTTP/1.1" 200 4 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" 工作时将replicas更改为1。但我认为这不是一个好的解决方案。我需要 3个副本。 

app-deployment.yaml

如何使其与3个副本一起使用？

1 个答案:

答案 0 :(得分：7)

在我更新Kubernetes服务模板以使用sessionAffinity: ClientIP后，它现在可以使用了。但是在第一次按下Ctrl + F5时会出现一些错误，而在第二次按下它时工作正常。

Error during WebSocket handshake: Unexpected response code: 400

但是，我仍然从服务器获取数据。所以我认为没关系。

更新了服务模板 

apiVersion: v1
kind: Service
metadata:
  name: app-nginx
  labels:
    app: app-nginx
spec:
  sessionAffinity: ClientIP
  type: LoadBalancer
  ports:
    # The port that this service should serve on.
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
    - port: 443
      targetPort: 443
      protocol: TCP
      name: https
  # Label keys and values that must match in order
  # to receive traffic for this service.
  selector:
    app: app-nginx
相关问题
最新问题