我有一个多线程程序,在所有线程中共享一个Engine
实例。它使用SQLAlchemy来访问MySQL以及Redshift。我有一个DbProvider
类用于获取Engine
个实例:(它是MySQL和Redshift的子类)
def get_db_engine(self) -> Engine:
if not self._engine:
self._engine = create_engine(self._get_connection(), pool_size=_POOL_SIZE, max_overflow=_MAX_OVERFLOW,
connect_args={'sslmode': 'verify-ca'})
# verify-ca to avoid certificate domain mismatch error when CNAME forwarding to AWS endpoint
return self._engine
该程序随机挂起,我无法找到重现此行为的方法。然而,经过很多努力,我发现它挂起的两个地方(哪一个发生取决于输入参数):
url = db_provider.get_db_engine().url
或
db_provider.get_db_engine().execute(script)
其中db_provider
是DbProvider
的实例,script
是字符串。
我的理解是Engine
是线程安全的。我将为每个实例尝试单独的Engine
,但这种情况发生得如此随机且几乎不能让我无法休息直到我知道真正的问题是什么。此外,单独的引擎也意味着单独的连接池可能会有性能损失。
更新:程序似乎挂起create_engine
。另外,我有时会在日志中收到此错误:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__
req = REQUIREMENT.parseString(requirement_string)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1617, in parseString
raise exc
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1607, in parseString
loc, tokens = self._parse( instring, 0 )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3376, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3698, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3359, in parseImpl
loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 2670, in parseImpl
raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 19), (line:1, col:20)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2998, in __init__
super(Requirement, self).__init__(requirement_string)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
requirement_string[e.loc:e.loc + 8]))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "''"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/redshift_copier/vertical_copier.py", line 65, in _run_table_copier
self._table_copier.copy(table)
File "/app/redshift_copier/table_copier.py", line 62, in copy
self._s3_to_redshift(table)
File "/app/redshift_copier/table_copier.py", line 147, in _s3_to_redshift
copier = S3RedshiftCopier(self._redshift_db_provider, s3_resource, self._redshift_copy_role)
File "/app/redshift_copier/s3_redshift_copier.py", line 21, in __init__
self._logger = Logger(obj=self, extra={log.FIELD_DATABASE: db_provider.get_db_engine().url})
File "/app/redshift_copier/db_provider.py", line 96, in get_db_engine
connect_args={'sslmode': 'verify-ca'})
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 56, in create
entrypoint = u._get_entrypoint()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/url.py", line 139, in _get_entrypoint
cls = registry.load(name)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 214, in load
return impl.load()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2405, in load
return self.resolve()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2411, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy_redshift/__init__.py", line 3, in <module>
__version__ = get_distribution('sqlalchemy-redshift').version
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 562, in get_distribution
dist = Requirement.parse(dist)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3044, in parse
req, = parse_requirements(s)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2991, in parse_requirements
yield Requirement(line)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3000, in __init__
raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "''"