Question

初始化程序抛出错误如下所示，脚本不会停止。
我想在开始主要流程之前中止（不要运行＆＃39; do_something＆＃39;）。

from multiprocessing import Pool
import contextlib

def initializer():
    raise Exception("init failed")

def do_something(args):
    # main process
    pass

pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
    try:
        pool.map_async(do_something, [1]).get(100)
    except:
        pool.terminate()

控制台上永不停止的堆栈跟踪在

之下

...
Exception: init failed
Process ForkPoolWorker-18:
Traceback (most recent call last):
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/pool.py", line 103, in worker
    initializer(*initargs)
  File "hoge.py", line 5, in initializer
    raise Exception("init failed")
Exception: init failed
...

我的解决方法是抑制初始化程序错误，并使用如下所示的全局标记在主进程开始时返回但我想学得更好。

def initializer():
    try:
        raise Exception("init failed")
    except:
        global failed
        failed = True

def do_something(args):
    global failed
    if failed:
        # skip when initializer failed
        return
    # main process

Answer 1

在使用PyCharm导航多处理的实现后，我确信没有更好的解决方案，因为如果存在任何工作进程，Pool会通过_repopulate_pool（）启动一个到_maintain_pool（）的线程 - 无论是意外还是失败初始化。

检查出来：Lib/multiprocessing/pool.py line 244

Answer 2

我刚遇到同样的祸患。我的第一个解决方案是捕获异常并在worker函数中引发它（见下文）。但第二个想法确实意味着initializer对multiprocessing.Pool的支持被破坏而且不被使用。所以我现在更喜欢直接在worker中进行初始化。

from multiprocessing import Pool
import contextlib, sys

_already_inited = False
def initializer():
    global _already_inited
    if _already_inited:
        return
    _already_inited = True
    raise Exception("init failed")

def do_something(args):
    initializer()
    # main process

pool = Pool(1)
with contextlib.closing(pool):
    pool.map_async(do_something, [1]).get(100)

代码和堆栈跟踪都比较简单。当然，您的所有工作人员都需要致电initializer()。

我最初的解决方案是将异常推迟到worker函数。

from multiprocessing import Pool
import contextlib, sys

failed = None
def initializer():
    try:
        raise Exception("init failed")
    except:
        global failed
        failed = sys.exc_info()[1]

def do_something(args):
    global failed
    if failed is not None:
        raise RuntimeError(failed) from failed
    # main process

pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
    pool.map_async(do_something, [1]).get(100)

这样调用者仍然可以访问异常。

multiprocessing.pool.RemoteTraceback:  
""" 
Traceback (most recent call last): 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker 
    result = (True, func(*args, **kwds)) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar 
    return list(map(*args)) 
  File "/tmp/try.py", line 15, in do_something 
    raise RuntimeError(failed) 
RuntimeError: init failed 
""" 

The above exception was the direct cause of the following exception: 

Traceback (most recent call last): 
  File "/tmp/try.py", line 20, in <module> 
    pool.map_async(do_something, [1]).get(100) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get 
    raise self._value 
RuntimeError: init failed 
(venv) kmkaplan@dev1:~/src/options$ python3 /tmp/try.py  
multiprocessing.pool.RemoteTraceback:  
""" 
Traceback (most recent call last): 
  File "/tmp/try.py", line 7, in initializer 
    raise Exception("init failed") 
Exception: init failed 

The above exception was the direct cause of the following exception: 

Traceback (most recent call last): 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker 
    result = (True, func(*args, **kwds)) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar 
    return list(map(*args)) 
  File "/tmp/try.py", line 15, in do_something 
    raise RuntimeError(failed) from failed 
RuntimeError: init failed 
""" 

The above exception was the direct cause of the following exception: 

Traceback (most recent call last): 
  File "/tmp/try.py", line 20, in <module> 
    pool.map_async(do_something, [1]).get(100) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get 
    raise self._value 
RuntimeError: init failed

如何在multiprocessing.Pool中处理初始化程序错误？

2 个答案: