Question

我得到了一个Python（2.7）函数，它接受3个字符串作为参数，并返回一个字典列表。由于项目的性质，我无法改变功能，这是非常复杂的，调用其他几个非标准的Python模块并使用psychopg2查询PostgreSQL数据库。我认为这是导致我出现问题的Postgres功能。

我想使用多处理模块来加速调用该函数数百次。我编写了一个“帮助器”函数，以便我可以使用multiprocessing.Pool（只需要1个参数）和我的函数：

from function_script import function

def function_helper(args):
    return function(*args)

我的主要代码如下：

from helper_script import function_helper
from multiprocessing import Pool

argument_a = ['a0', 'a1', ..., 'a99']
argument_b = ['b0', 'b1', ..., 'b99']
argument_c = ['c0', 'c1', ..., 'c99']

input = zip(argument_a, argument_b, argument_c)
p = Pool(4)
results = p.map(function_helper, input)

print results

我期待的是字典列表的列表，但是我收到以下错误：

Traceback (most recent call last):
  File "/local/python/2.7/lib/python2.7/site-packages/variantValidator/variantValidator.py", line 898, in validator
    vr.validate(input_parses)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 33, in validate
    return self._ivr.validate(var, strict) and self._evr.validate(var, strict)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 69, in validate
    (res, msg) = self._ref_is_valid(var)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 89, in _ref_is_valid
    var_x = self.vm.c_to_n(var) if var.type == "c" else var
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 223, in c_to_n
    tm = self._fetch_TranscriptMapper(tx_ac=var_c.ac, alt_ac=var_c.ac, alt_aln_method="transcript")
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
    result = user_function(*args, **kwds)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 372, in _fetch_TranscriptMapper
    self.hdp, tx_ac=tx_ac, alt_ac=alt_ac, alt_aln_method=alt_aln_method)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/transcriptmapper.py", line 69, in __init__
    self.tx_identity_info = hdp.get_tx_identity_info(self.tx_ac)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
    result = user_function(*args, **kwds)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 353, in get_tx_identity_info
    rows = self._fetchall(self._queries['tx_identity_info'], [tx_ac])
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 216, in _fetchall
    with self._get_cursor() as cur:
  File "/local/python/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 529, in _get_cursor
    cur.execute("set search_path = " + self.url.schema + ";")
  File "/local/python/2.7/lib/python2.7/site-packages/psycopg2/extras.py", line 144, in execute
    return super(DictCursor, self).execute(query, vars)
DatabaseError: SSL error: decryption failed or bad record mac

和

Traceback (most recent call last):
  File "/local/python/2.7/lib/python2.7/site-packages/variantValidator/variantValidator.py", line 898, in validator
    vr.validate(input_parses)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 33, in validate
    return self._ivr.validate(var, strict) and self._evr.validate(var, strict)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 69, in validate
    (res, msg) = self._ref_is_valid(var)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 89, in _ref_is_valid
    var_x = self.vm.c_to_n(var) if var.type == "c" else var
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 223, in c_to_n
    tm = self._fetch_TranscriptMapper(tx_ac=var_c.ac, alt_ac=var_c.ac, alt_aln_method="transcript")
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
    result = user_function(*args, **kwds)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 372, in _fetch_TranscriptMapper
    self.hdp, tx_ac=tx_ac, alt_ac=alt_ac, alt_aln_method=alt_aln_method)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/transcriptmapper.py", line 69, in __init__
    self.tx_identity_info = hdp.get_tx_identity_info(self.tx_ac)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
    result = user_function(*args, **kwds)
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 353, in get_tx_identity_info
    rows = self._fetchall(self._queries['tx_identity_info'], [tx_ac])
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 216, in _fetchall
    with self._get_cursor() as cur:
  File "/local/python/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 526, in _get_cursor
    conn.autocommit = True
InterfaceError: connection already closed

有没有人知道什么可能导致Pool函数表现得像这样，在我试过的其他例子中使用它似乎很简单？如果这还没有足够的信息继续下去，任何人都可以建议我找到问题的根源（这是我第一次使用其他人的代码）？或者，有没有其他方法可以使用多处理模块调用该函数数百次？

由于

Answer 1

我认为可能发生的情况是，您的连接对象将在所有工作人员中使用，并且当一位工作人员完成所有任务时，它将关闭连接，而其他工作人员仍在工作并且关闭连接，因此当其中一个工作人员尝试使用已关闭的数据库。

查询PostgreSQL数据库的黑盒函数使用multiprocessing.Pool时，“InterfaceError：连接已经关闭”

1 个答案: