SQLAlchemy引擎挂起

时间:2018-04-06 06:40:33

标签: python sqlalchemy

我有一个多线程程序,在所有线程中共享一个Engine实例。它使用SQLAlchemy来访问MySQL以及Redshift。我有一个DbProvider类用于获取Engine个实例:(它是MySQL和Redshift的子类)

def get_db_engine(self) -> Engine:
    if not self._engine:
        self._engine = create_engine(self._get_connection(), pool_size=_POOL_SIZE, max_overflow=_MAX_OVERFLOW,
                                     connect_args={'sslmode': 'verify-ca'})
        # verify-ca to avoid certificate domain mismatch error when CNAME forwarding to AWS endpoint

    return self._engine

该程序随机挂起,我无法找到重现此行为的方法。然而,经过很多努力,我发现它挂起的两个地方(哪一个发生取决于输入参数):

url = db_provider.get_db_engine().url

db_provider.get_db_engine().execute(script)

其中db_providerDbProvider的实例,script是字符串。

我的理解是Engine是线程安全的。我将为每个实例尝试单独的Engine,但这种情况发生得如此随机且几乎不能让我无法休息直到我知道真正的问题是什么。此外,单独的引擎也意味着单独的连接池可能会有性能损失。

更新:程序似乎挂起create_engine。另外,我有时会在日志中收到此错误:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1617, in parseString
    raise exc
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1607, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3376, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3698, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3359, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 2670, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 19), (line:1, col:20)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2998, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "''"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/redshift_copier/vertical_copier.py", line 65, in _run_table_copier
    self._table_copier.copy(table)
  File "/app/redshift_copier/table_copier.py", line 62, in copy
    self._s3_to_redshift(table)
  File "/app/redshift_copier/table_copier.py", line 147, in _s3_to_redshift
    copier = S3RedshiftCopier(self._redshift_db_provider, s3_resource, self._redshift_copy_role)
  File "/app/redshift_copier/s3_redshift_copier.py", line 21, in __init__
    self._logger = Logger(obj=self, extra={log.FIELD_DATABASE: db_provider.get_db_engine().url})
  File "/app/redshift_copier/db_provider.py", line 96, in get_db_engine
    connect_args={'sslmode': 'verify-ca'})
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 56, in create
    entrypoint = u._get_entrypoint()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/url.py", line 139, in _get_entrypoint
    cls = registry.load(name)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 214, in load
    return impl.load()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2405, in load
    return self.resolve()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2411, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy_redshift/__init__.py", line 3, in <module>
    __version__ = get_distribution('sqlalchemy-redshift').version
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 562, in get_distribution
    dist = Requirement.parse(dist)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3044, in parse
    req, = parse_requirements(s)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2991, in parse_requirements
    yield Requirement(line)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3000, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "''"

0 个答案:

没有答案