Question

我的Python项目中有一个奇怪的问题。它使用SQLAlchemy和Bakery来准备查询。我有一个需要连接（db），面包店和一系列对象的函数。

此函数在for循环中被其他函数多次调用，这是我的问题（至少我了解）：

让我们假设它第一次接收到一个包含两个元素的数组。
下次调用该函数时，也会期望包含两个元素的数组

import sqlalchemy as sa
def cpe_filter(db, bakery, iterable):
    cpes = []

    try:
        query  = bakery(lambda s: s.query(Cpe))
        query += lambda y: y.filter(
            sa.or_(*[
                Cpe.cpe.like(sa.bindparam('cpe_{}'.format(i)))
                for i, _ in enumerate(iterable)
            ])
        )
        query += lambda y: y.filter_by(active=sa.bindparam('active'))

        cpes = query(db).params(active=True,
                                **{'cpe_{}'.format(i): e for i, e in enumerate(iterable)}) \
                        .all()
    except NoResultFound:
        log.info("Found no CPE matching list {}.".format(iterable))

如果下一个数组比前一个数组小，则会出现这种错误（Pastebin）：

[2018-12-17 16:35:16 - INFO/sqlalchemy.engine.base.Engine:1151] SELECT cpe.id AS cpe_id, cpe.active AS cpe_active, cpe.date_created AS cpe_date_created, cpe.timestamp AS cpe_timestamp, cpe.cpe_part_id AS cpe_cpe_part_id, cpe.device_id AS cpe_device_id, cpe.cpe AS cpe_cpe, cpe.match_nvd AS cpe_match_nvd
FROM cpe
WHERE (cpe.cpe LIKE %(cpe_0)s OR cpe.cpe LIKE %(cpe_1)s OR cpe.cpe LIKE %(cpe_2)s) AND cpe.active = %(active)s
[2018-12-17 16:35:16 - INFO/sqlalchemy.engine.base.Engine:1154] {'cpe_0': 'cpe:/o:sun:solaris', 'cpe_1': 'cpe:/a:tritreal:ted_cde', 'cpe_2': 'cpe:/o:hp:hp-ux', 'active': 1}
[2018-12-17 16:35:16 - INFO/sqlalchemy.engine.base.Engine:1151] SELECT cpe.id AS cpe_id, cpe.active AS cpe_active, cpe.date_created AS cpe_date_created, cpe.timestamp AS cpe_timestamp, cpe.cpe_part_id AS cpe_cpe_part_id, cpe.device_id AS cpe_device_id, cpe.cpe AS cpe_cpe, cpe.match_nvd AS cpe_match_nvd
FROM cpe
WHERE (cpe.cpe LIKE %(cpe_0)s OR cpe.cpe LIKE %(cpe_1)s OR cpe.cpe LIKE %(cpe_2)s) AND cpe.active = %(active)s
[2018-12-17 16:35:16 - INFO/sqlalchemy.engine.base.Engine:1154] {'cpe_0': 'cpe:/a:hp:dtmail', 'cpe_1': 'cpe:/a:university_of_washington:pine', 'cpe_2': 'cpe:/o:sco:unixware', 'active': 1}
[2018-12-17 16:35:16 - ERROR/scap.abc:66] An error has occurred during task execution.
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_context
    context = constructor(dialect, self, conn, *args)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 635, in _init_compiled
    grp, m in enumerate(parameters)]
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 635, in <listcomp>
    grp, m in enumerate(parameters)]
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 547, in construct_params
    % bindparam.key, code="cd3x")
sqlalchemy.exc.InvalidRequestError: A value is required for bind parameter 'cpe_2' (Background on this error at: http://sqlalche.me/e/cd3x)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/scap/project/scap/abc.py", line 64, in run
    self(*args, **kwargs)
  File "/root/scap/project/scap/tasks.py", line 362, in __call__
    q.cve_insert_or_update(self.db, self.bakery, self.parse(name))
  File "/root/scap/project/scap/queries.py", line 148, in cve_insert_or_update
    cpes = list(cpe_filter(db, bakery, cpes))
  File "/root/scap/project/scap/queries.py", line 68, in cpe_filter
    **{'cpe_{}'.format(i): e for i, e in enumerate(products)}) \
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 457, in all
    return list(self)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 364, in __iter__
    return q._execute_and_instances(context)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3018, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context
    None, None)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_context
    context = constructor(dialect, self, conn, *args)
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 635, in _init_compiled
    grp, m in enumerate(parameters)]
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 635, in <listcomp>
    grp, m in enumerate(parameters)]
  File "/root/.local/share/virtualenvs/scap-TS2Ah8Sl/lib/python3.6/site-packages/sqlalchemy/sql/compiler.py", line 547, in construct_params
    % bindparam.key, code="cd3x")
sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A value is required for bind parameter 'cpe_2' [SQL: 'SELECT cpe.id AS cpe_id, cpe.active AS cpe_active, cpe.date_created AS cpe_date_created, cpe.timestamp AS cpe_timestamp, cpe.cpe_part_id AS cpe_cpe_part_id, cpe.device_id AS cpe_device_id, cpe.cpe AS cpe_cpe, cpe.match_nvd AS cpe_match_nvd \nFROM cpe \nWHERE (cpe.cpe LIKE %(cpe_0)s OR cpe.cpe LIKE %(cpe_1)s OR cpe.cpe LIKE %(cpe_2)s) AND cpe.active = %(active)s'] [parameters: [{'active': True, 'cpe_0': 'cpe:/a:university_of_washington:imap', 'cpe_1': 'cpe:/a:netscape:messaging_server'}]] (Background on this error at: http://sqlalche.me/e/cd3x)

如您所见，该函数被调用了三次，前两次它没有问题地工作（每次3个元素），第三次它只有两个元素，并且根据错误期望第三个元素

NB：大多数情况下，可迭代项可以达到约50个元素。

Answer 1

问题源于烘焙查询文档中"Synopsis"下的观察值4.和5。：

在上面的代码中，即使我们的应用程序可能多次调用search_for_user()，并且即使在每次调用中我们建立了一个全新的BakedQuery对象，所有的lambda都是只能调用一次。只要此查询被缓存在面包店中，每个Lambda都不会被再次调用。

通过存储对 lambda对象本身的引用来制定缓存键，从而实现缓存；也就是说，Python解释器为这些函数分配了Python内标识，这一事实决定了如何在连续运行中识别查询。对于指定了search_for_user()参数的email的调用，可调用的lambda q: q.filter(User.email == bindparam('email'))将成为所检索的缓存键的一部分；当email为None时，此可调用项不属于缓存键。

如果使用dis检查cpe_filter()函数，您会注意到lambda函数是常量，因此在调用之间保持其身份。如参考文档中所述，SQLAlchemy根据这些标识和调用来缓存查询

query += lambda y: y.filter(
    sa.or_(*[
        Cpe.cpe.like(sa.bindparam('cpe_{}'.format(i)))
        for i, _ in enumerate(iterable)
    ])
)

仅一次。换句话说，占位符将在您第一次调用cpe_filter()时基于iterable进行设置。仅当从缓存中清除此查询时，它们才会“重置”。

该解决方案取决于您使用的DBMS。例如Postgresql具有ANY数组比较可以使用：

query += lambda y: y.filter(Cpe.cpe.like(sa.any_(sa.bindparam('cpe'))))

，参数将作为

传递

# This relies on Psycopg2's lists adaptation:
# http://initd.org/psycopg/docs/usage.html#lists-adaptation
cpes = query(db).params(active=True, cpe=list(iterable)).all()

在MS SQL Server上，您可以创建一个full-text index并使用CONTAINS：

query += lambda y: y.filter(func.contains(Cpe.cpe, sa.bindparam('cpe')))

绑定参数cpe应该通过搜索条件，该条件必须由iterable组成：

search_cond = " OR ".join(iterable)
cpes = query(db).params(active=True, cpe=search_cond).all()

这当然要求iterable中的项目是有效的全文搜索词。

SQLAlchemy Bakery-函数需要一定的数组大小

1 个答案: