Question

我使用Pool模块的multiprocessing函数，以便在不同数据上并行运行相同的代码。

事实证明，在某些数据中，我的代码会引发异常，但是没有给出发生这种情况的确切行：

Traceback (most recent call last):
  File "my_wrapper_script.py", line 366, in <module>
    main()
  File "my_wrapper_script.py", line 343, in main
    results = pool.map(process_function, folders)
  File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
    raise self._value
KeyError: 'some_key'

我知道multiprocessing.log_to_stderr()，但似乎在出现并发问题时它很有用，这不是我的情况。

有什么想法吗？

Answer 1

如果您使用的是足够新版本的Python，那么您实际上会看到真正的异常在此之前被打印出来。例如，这里有一个失败的样本：

import multiprocessing

def inner():
    raise Exception("FAIL")

def f():
    print("HI")
    inner()

p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()

这是使用python 3.4运行它时的例外：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "test.py", line 9, in f
    inner()
  File "test.py", line 4, in inner
    raise Exception("FAIL")
Exception: FAIL
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    p.apply(f)
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 253, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
Exception: FAIL

如果使用较新的版本不是一个选项，最简单的方法是将您的worker函数包装在try / except块中，该块将在重新提升之前打印异常：

import multiprocessing
import traceback

def inner():
    raise Exception("FAIL")

def f():
    try:
        print("HI")
        inner()
    except Exception:
        print("Exception in worker:")
        traceback.print_exc()
        raise

p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()

输出：

HI
Exception in worker:
Traceback (most recent call last):
  File "test.py", line 11, in f
    inner()
  File "test.py", line 5, in inner
    raise Exception("FAIL")
Exception: FAIL
Traceback (most recent call last):
  File "test.py", line 18, in <module>
    p.apply(f)
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 244, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
Exception: FAIL

Answer 2

您需要在worker中实现自己的try / except块。根据您希望如何组织代码，您可以如上所述登录到stderr，登录其他地方（如文件），返回某种错误代码，甚至使用当前回溯标记异常并重新加注。以下是最后一种技术的示例：

import traceback
import multiprocessing as mp

class MyError(Exception):
    pass

def worker():
    try:
        # your real code here
        raise MyError("boom")
    except Exception, e:
        e.traceback = traceback.format_exc()
        raise

def main():
    pool = mp.Pool()
    try:
        print "run worker"
        result = pool.apply_async(worker)
        result.get()
    # handle exceptions you expect
    except MyError, e:
        print e.traceback
    # re-raise the rest
    except Exception, e:
        print e.traceback
        raise


if __name__=="__main__":
    main()

返回

run worker
Traceback (most recent call last):
  File "doit.py", line 10, in worker
    raise MyError("boom")
MyError: boom

调试python多处理中的错误

2 个答案: