Question

我在GKE中有3个Pod的gRPC服务。每个吊舱都已附加带有rollout_strategy = managed的ESP容器，后端指向gRPC。

有时候我碰到http端点

{
 "code": 13,
 "message": "Failed to fetch service account token",
 "details": [
  {
   "@type": "type.googleapis.com/google.rpc.DebugInfo",
   "stackEntries": [],
   "detail": "internal"
  }
 ]
}

每三个请求失败。当我删除每个Pod时，它开始起作用。我认为一小时后令牌将失效。

部署ESP容器

 containers:
      - args:
        - --http_port=80
        - --backend=grpc://127.0.0.1:6000
        - --service=service.endpoints.cluster.cloud.goog
        - --rollout_strategy=managed
        image: gcr.io/endpoints-release/endpoints-runtime:1
        imagePullPolicy: IfNotPresent
        name: esp
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File

在容器日志中，存在如下错误：

2019/07/17 07:51:05[error]10#10: Failed to call https://servicecontrol.googleapis.com/v1/services/service.endpoints.cluster.cloud.goog:report, Error: UNAUTHORIZED: server response status code: 401, Response body: �Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
[libprotobuf ERROR external/servicecontrol_client_git/src/service_control_client_impl.cc:182] Failed in Report call: Service control request failed with HTTP response code 401

我在这里https://groups.google.com/forum/#!topic/google-cloud-endpoints/0EEY88X8oIY发现了相同的问题但这只是通过显式提供服务帐户来解决。

编辑：我尝试使用服务帐户，但是pod尝试刷新令牌时仍然存在问题

2019-07-19T11:45:48.031715293Z INFO:Refreshing access_token
 E 
2019-07-19T11:45:48.071914451Z Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 365, in execute
  File ".bootstrap/_pex/pex.py", line 293, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 325, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 408, in _execute
  File ".bootstrap/_pex/pex.py", line 466, in execute_entry
  File ".bootstrap/_pex/pex.py", line 471, in execute_module
  File "/usr/lib/python2.7/runpy.py", line 192, in run_module
    fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "start_esp/start_esp.py", line 1061, in <module>
  File "start_esp/start_esp.py", line 343, in fetch_service_config
  File "start_esp/fetch_service_config.py", line 151, in make_access_token
  File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 663, in get_access_token
 E 
2019-07-19T11:45:48.074420362Z     self.refresh(http)
 E 
2019-07-19T11:45:48.074424955Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 545, in refresh
 E 
2019-07-19T11:45:48.074430139Z     self._refresh(http)
 E 
2019-07-19T11:45:48.074434982Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 749, in _refresh
 E 
2019-07-19T11:45:48.074440214Z     self._do_refresh_request(http)
 E 
2019-07-19T11:45:48.074444687Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/client.py", line 780, in _do_refresh_request
 E 
2019-07-19T11:45:48.074450391Z     body=body, headers=headers)
 E 
2019-07-19T11:45:48.074455280Z   File "/root/.pex/install/oauth2client-4.1.3-py2.py3-none-any.whl.289a6efc12c558f2f0ae1841479ec3ee7a783c1b/oauth2client-4.1.3-py2.py3-none-any.whl/oauth2client/transport.py", line 282, in request
 E 
2019-07-19T11:45:48.074460859Z     connection_type=connection_type)
 E 
2019-07-19T11:45:48.074465988Z   File "/root/.pex/install/httplib2-0.13.0-py2-none-any.whl.f7b2ff9b447b5741fa19aab52032ab1c7db17f35/httplib2-0.13.0-py2-none-any.whl/httplib2/__init__.py", line 2135, in request
 E 
2019-07-19T11:45:48.074482470Z     cachekey,
 E 
2019-07-19T11:45:48.074487590Z   File "/root/.pex/install/httplib2-0.13.0-py2-none-any.whl.f7b2ff9b447b5741fa19aab52032ab1c7db17f35/httplib2-0.13.0-py2-none-any.whl/httplib2/__init__.py", line 1796, in _request
 E 
2019-07-19T11:45:48.074492698Z     conn, request_uri, method, body, headers
 E 
2019-07-19T11:45:48.074497648Z   File "/root/.pex/install/httplib2-0.13.0-py2-none-any.whl.f7b2ff9b447b5741fa19aab52032ab1c7db17f35/httplib2-0.13.0-py2-none-any.whl/httplib2/__init__.py", line 1737, in _conn_request
 E 
2019-07-19T11:45:48.074502516Z     response = conn.getresponse()
 E 
2019-07-19T11:45:48.074506952Z   File "/usr/lib/python2.7/httplib.py", line 1108, in getresponse
 E 
2019-07-19T11:45:48.074511855Z     raise ResponseNotReady()
 E 
2019-07-19T11:45:48.074516567Z httplib.ResponseNotReady
 E

此后，pod重新启动，一切正常1小时。

Answer 1

GCP具有用于ESP的元数据服务器，以获取访问令牌。如果超载，ESP可能会失败，因为它需要访问令牌才能与Google服务对话。如果此问题经常发生，则应向GKE团队提交错误。

其他用户也遇到了交通拥挤的类似情况。我向他们建议了与https://groups.google.com/forum/#!topic/google-cloud-endpoints/0EEY88X8oIY

相同的解决方法

GKE ESP无法获取服务帐户令牌

1 个答案: