将Google Cloud KMS与Flask,gunicorn和gevent worker结合使用时出错:“ google.api_core.exceptions.ServiceUnavailable:超过503个截止日期”

时间:2019-06-22 20:56:45

标签: google-app-engine flask gunicorn gevent

在Flask请求的上下文中尝试使用google-cloud-kms进行加密或解密时,向带有gunicorn和gevent工作者的Flask应用提供服务时会出现以下错误。无论是在本地运行应用还是在Google App Engine(灵活)环境中,都会发生该错误。

app_1  | Traceback (most recent call last):
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
app_1  |     return callable_(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
app_1  |     return _end_unary_response_blocking(state, call, False, None)
app_1  |   File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
app_1  |     raise _Rendezvous(state, None, None, deadline)
app_1  | grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
app_1  |    status = StatusCode.UNAVAILABLE
app_1  |    details = "Deadline Exceeded"
app_1  |    debug_error_string = "{"created":"@1561236057.820157100","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":14}"
app_1  | >
app_1  |
app_1  | The above exception was the direct cause of the following exception:
app_1  |
app_1  | Traceback (most recent call last):
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
app_1  |     response = self.full_dispatch_request()
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
app_1  |     rv = self.handle_user_exception(e)
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
app_1  |     reraise(exc_type, exc_value, tb)
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
app_1  |     raise value
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
app_1  |     rv = self.dispatch_request()
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
app_1  |     return self.view_functions[rule.endpoint](**req.view_args)
app_1  |   File "/app/app.py", line 14, in encrypt
app_1  |     encrypted = client.encrypt(key_name, plaintext.encode())
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/cloud/kms_v1/gapic/key_management_service_client.py", line 1286, in encrypt
app_1  |     request, retry=retry, timeout=timeout, metadata=metadata
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
app_1  |     return wrapped_func(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
app_1  |     on_error=on_error,
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
app_1  |     return target()
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
app_1  |     return func(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
app_1  |     six.raise_from(exceptions.from_grpc_error(exc), exc)
app_1  |   File "<string>", line 3, in raise_from
app_1  | google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded

requirements.txt

Flask==1.0.3
gevent==1.4.0
google-cloud-kms==1.0.0
grpcio==1.21.1
gunicorn==19.9.0

复制此示例的示例Flask应用:

from flask import Flask
from google.cloud import kms_v1
import base64


def create_app():
    app = Flask(__name__)

    @app.route('/')
    def encrypt():
        plaintext = "plain"
        client = kms_v1.KeyManagementServiceClient()
        key_name = client.crypto_key_path('project', 'location', 'keyring', 'key')
        encrypted = client.encrypt(key_name, plaintext.encode())
        return base64.urlsafe_b64encode(encrypted.ciphertext)

    return app


example_app = create_app()

搭配:

gunicorn -t 300 -b :8080 -k gevent -w 4 app:example_app

1 个答案:

答案 0 :(得分:1)

在花一些时间调试此问题时回答了我自己的问题,并在网上找到了类似但不是特定问题的答案。

问题似乎是grpc和gevent如何一起工作(或不工作),而最快的解决方法是使用另一名枪手工人。经过测试,至少同步,eventlet和gthread工作程序阻止了此错误的发生。

gunicorn -t 300 -b :8080 -k eventlet -w 4 app:example_app