Question

我正在尝试创建一个python装饰器，该装饰器使用带有args和kwargs的函数，在新进程中执行该函数，将其关闭，并返回返回的任何函数，包括引发相同的异常（如果有）。

就目前而言，我的装饰器可以正常处理函数，只要它们不引发异常，但无法提供回溯。如何将其传递回父进程？

from functools import wraps
from multiprocessing import Process, Queue
import sys


def process_wrapper(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        # queue for communicating between parent and child processes
        q = Queue()

        def func_to_q(_q: Queue, *_args, **_kwargs):
            # do the same as func, but put result into the queue. Also put
            # there an exception if any.
            try:
                _res = func(*_args, **_kwargs)
                _q.put(_res)
            except:
                _q.put(sys.exc_info())

        # start another process and wait for it to join
        p = Process(target=func_to_q, args=(q, )+args, kwargs=kwargs)
        p.start()
        p.join()

        # get result from the queue and return it, or raise if it's an exception
        res = q.get(False)
        if isinstance(res, tuple) and isinstance(res[0], Exception):
            raise res[1].with_traceback(res[2])
        else:
            return res
    return wrapper


if __name__ == '__main__':

    @process_wrapper
    def ok():
        return 'ok'

    @process_wrapper
    def trouble():
        def inside():
            raise UserWarning
        inside()

    print(ok())
    print(trouble())

我希望结果像这样：

ok
Traceback (most recent call last):
  File "/temp.py", line 47, in <module>
    print(trouble())
  File "/temp.py", line 44, in trouble
    inside()
  File "/temp.py", line 43, in inside
    raise UserWarning
UserWarning

Process finished with exit code 1

但是似乎子进程无法将stacktrace放入队列，并且我得到以下信息：

ok
Traceback (most recent call last):
  File "/temp.py", line 47, in <module>
    print(trouble())
  File "/temp.py", line 26, in wrapper
    res = q.get(False)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 107, in get
    raise Empty
queue.Empty

Process finished with exit code 1

此外，如果子进程仅将异常本身_q.put(sys.exc_info()[1])放入队列，父进程从那里获取异常并引发异常，但带有新的堆栈跟踪（注意缺少对inside()的调用）：

ok
Traceback (most recent call last):
  File "/temp.py", line 47, in <module>
    print(trouble())
  File "/temp.py", line 28, in wrapper
    raise res
UserWarning

Process finished with exit code 1

Answer 1

看看multiprocessing.pool.py和用于将异常发送给父级的stringification-hack。您可以从那里使用multiprocessing.pool.ExceptionWithTraceback。

这足以说明基本原理：

from multiprocessing import Process, Queue
from multiprocessing.pool import ExceptionWithTraceback


def worker(outqueue):
    try:
        result = (True, 1 / 0)  # will raise ZeroDivisionError
    except Exception as e:
        e = ExceptionWithTraceback(e, e.__traceback__)
        result = (False, e)
    outqueue.put(result)

if __name__ == '__main__':

    q = Queue()
    p = Process(target=worker, args=(q,))
    p.start()
    success, value = q.get()
    p.join()

    if success:
        print(value)
    else:
        raise value  # raise again

输出：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/...", line 7, in worker
    result = (True, 1 / 0)  # will raise ZeroDivisionError
ZeroDivisionError: division by zero
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/...", line 23, in <module>
    raise value
ZeroDivisionError: division by zero

Process finished with exit code 1

如何在Python的进程之间传递stacktrace？

1 个答案: