Question

我正在尝试并行执行任务，并决定使用多处理（mp）模块。我很快了解到mp无法处理嵌套函数，因此我必须取消嵌套与此类似的函数：

def foo(nomer1, nomer2):

    def bar(nomer1, nomer2):
        return ([nomer1] * nomer2)

    return(bar(nomer1, nomer2))

对此：

def bar(nomer1, nomer2):
    return([nomer1]*nomer2)

def foo(nomer1, nomer2):
    return(bar(nomer1, nomer2))

但是，当我跑步时：

import multiprocessing as mp
n_proc = 2
pool = mp.Pool(processes=n_proc)
pool.starmap(foo, [11, 2])

控制台产生此错误：

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\user\Anaconda2\envs\env1\lib\multiprocessing\pool.py", line 274, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "C:\Users\user\Anaconda2\envs\env1\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "C:\Users\user\Anaconda2\envs\env1\lib\multiprocessing\pool.py", line 424, in _handle_tasks
    put(task)
  File "C:\Users\user\Anaconda2\envs\env1\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Users\user\Anaconda2\envs\env1\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function foo at 0x00000189D37902F0>: attribute lookup foo on __main__ failed

Foo是documentation中指定的最高级别，但是mp仍然无法对其进行酸洗。我该如何解决这个问题，最好不安装其他软件包？

Answer 1

我已经使用concurrent.futures解决了这个问题
该软件包应随我的Python安装一起提供，该问题的解决方法如下：

pool = ThreadPoolExecutor(2)
future = pool.submit(foo, 11,2)
print(future.result())

错误似乎是a known bug的结果。建议的Anaconda更新无济于事。我想知道在报告了一年之后该错误是否仍然存在。

与此同时，未来的ProcessPoolExecutor也会产生类似的错误：

Traceback (most recent call last):
  File "C:\Users\user\Anaconda2\envs\env1\lib\multiprocessing\queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "C:\Users\user\Anaconda2\envs\env1\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function foo at 0x000001D816CF5510>: attribute lookup foo on __main__ failed

多重处理无法使顶级功能发挥作用

1 个答案: