分叉,sqlalchemy和范围会话

时间:2017-02-09 04:31:51

标签: python sqlalchemy multiprocessing dask

我收到以下错误(我假设是因为我的应用程序中的分叉),“此结果对象不返回行”。

Traceback
---------
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/dask/async.py", line 263, in execute_task
result = _execute_task(task, data)
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/dask/async.py", line 245, in _execute_task
return func(*args2)
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/smg/analytics/services/impact_analysis.py", line 140, in _do_impact_analysis_mp
 Correlation.user_id.in_(user_ids)).all())
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2241, in all
return list(self)
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 65, in instances
fetch = cursor.fetchall()
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 752, in fetchall
self.cursor, self.context)
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1027, in _handle_dbapi_exception
util.reraise(*exc_info)
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 746, in fetchall
l = self.process_rows(self._fetchall_impl())
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 715, in _fetchall_impl
self._non_result()
File "/opt/miniconda/envs/analytical-engine/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 720, in _non_result
"This result object does not return rows. "

我正在使用dask和它的多处理调度程序(使用multiprocessing.Pool)。 根据我的理解(基于文档),从作用域会话对象(通过scoped_session()创建)创建的会话是线程安全的。这是因为它们是threadlocal。这会让我相信当我调用Session()(或使用代理Session)时,我会得到一个仅存在的会话对象,并且只能从调用它的线程访问。 这看起来非常简单。

我感到困惑的是,为什么我在分叉过程中遇到问题。我明白你不能 重复使用跨进程的引擎,所以我从文档中逐字逐句地遵循基于事件的解决方案并完成了这个:

class _DB(object):

    _engine = None

    @classmethod
    def _get_engine(cls, force_new=False):
        if cls._engine is None or force_new is True:
            cfg = Config.get_config()
            user = cfg['USER']
            host = cfg['HOST']
            password = cfg['PASSWORD']
            database = cfg['DATABASE']
            engine = create_engine(
                'mysql://{}:{}@{}/{}?local_infile=1&'
                'unix_socket=/var/run/mysqld/mysqld.sock'.
                    format(user, password, host, database),
                pool_size=5, pool_recycle=3600)
            cls._engine = engine
        return cls._engine



# From the docs, handles multiprocessing
@event.listens_for(_DB._get_engine(), "connect")
def connect(dbapi_connection, connection_record):
    connection_record.info['pid'] = os.getpid()

#From the docs, handles multiprocessing
@event.listens_for(_DB._get_engine(), "checkout")
def checkout(dbapi_connection, connection_record, connection_proxy):
    pid = os.getpid()
    if connection_record.info['pid'] != pid:
        connection_record.connection = connection_proxy.connection = None
        raise exc.DisconnectionError(
            "Connection record belongs to pid %s, "
            "attempting to check out in pid %s" %
            (connection_record.info['pid'], pid)
        )


# The following is how I create the scoped session object.

Session = scoped_session(sessionmaker(
    bind=_DB._get_engine(), autocommit=False, autoflush=False))

Base = declarative_base()
Base.query = Session.query_property()

所以我的假设(基于文档)如下:

  1. 使用从作用域会话对象创建的会话对象,它必须始终给我一个threadlocal会话(在我的情况下,它只是子进程的主线程)。虽然不在我想象的文档中,但即使在另一个进程中创建了作用域会话对象,这也应该适用。

  2. threadlocal会话将通过引擎从池中获得连接,如果在此过程中未创建连接,则会创建一个新连接(基于上面的connection()和{{1}实现。)

  3. 如果这些都是真的,那么一切都应该“正常”(AFAICT)。但事实并非如此。

    我设法通过在每个新进程中创建一个新的作用域会话对象并使用它来使其工作 在使用会话的所有后续呼叫中。

    BTW还需要从这个新的作用域会话对象更新checkout()属性。

    我想我上面的#1假设是不正确的。任何人都可以帮助我理解为什么我需要在每个进程中创建一个新的作用域会话对象吗?

    干杯。

1 个答案:

答案 0 :(得分:0)

目前尚不清楚fork何时发生,但最常见的问题是引擎是在fork之前创建的,它使用pool_size = 5初始化与数据库的TCP连接,然后将其复制到新进程和结果在多个进程中与相同的物理套接字进行交互=>的烦恼。

选项是:

  • 禁用池并使用按需连接:poolclass = NullPool
  • 在fork之后重新创建池:sqla_engine。dispose()
  • 将create_engine延迟到fork
  • 之后