下面是一个Python问题，演示了如何使用func并行迭代函数multiprocessing.Pool。要迭代的Np个元素数。函数func仅返回Np减去可迭代的索引。如图所示，当以并行模式运行时，我使用队列来返回函数中的值。
如果我设置runParallel=False，程序可以在串行模式下执行。
该程序运行良好，适用于runParallel=False和runParallel=True，但现在出现了我遇到的基本问题：如下所示，如果将problemIndex设置为低于{{1} （例如Np），然后我做一个浮点异常。我除以零 - 愚蠢的我： - ）
如果正在运行problemIndex=7，那么我可以看到错误的源代码行，我直接捕获了错误。
runParallel=False尼斯！
然而对于$ python map.py
Traceback (most recent call last):
File "map.py", line 63, in <module>
a = func(argList[p])
File "map.py", line 22, in func
ret = 1/(args["index"]-args["problemIndex"])
ZeroDivisionError: integer division or modulo by zero
我最终只是在＆＃34; Bummer＆＃34;打印部分没有错误来源的指示。讨厌！
我的问题是：对于runParallel=True，如何有效地调试此问题并从runParallel=True获取错误代码行的行号？
Pool()答案 0 :(得分：2)
我发现traceback模块在多处理调试中非常有用。如果将异常传递回主线程/进程，则会丢失所有回溯信息，因此您需要在子线程中调用traceback.format_exc并将该文本传递回主线程，但异常。下面我包括一个可以与Pool一起使用的模式。
import traceback
import multiprocessing as mp
import time
def mpFunctionReportError(kwargs):
'''
wrap any function and catch any errors from f,
putting them in pipe instead of raising
kwargs must contain 'queue' (multiprocessing queue)
and 'f' function to be run
'''
queue = kwargs.pop('queue')
f = kwargs.pop('f')
rslt=None
try:
rslt = f(**kwargs)
queue.put(rslt)
except Exception, e:
queue.put([e,traceback.format_exc(e)])
return
def doNothing(a):
return a
def raiseException(a):
a='argh'
raise ValueError('this is bad')
manager = mp.Manager()
outQ = manager.Queue()
p = mp.Pool(processes=4)
ret = p.map_async(mpFunctionReportError,iterable=[dict(f=doNothing,queue=outQ,a='pointless!') for i in xrange(4)])
ret.wait()
time.sleep(1)
for i in xrange(4):
print(outQ.get_nowait())
ret = p.map_async(mpFunctionReportError,iterable=[dict(f=raiseException,queue=outQ,a='pointless!') for i in xrange(2)])
ret.wait()
time.sleep(1)
for i in xrange(2):
e,trace = outQ.get_nowait()
print(e)
print(trace)
运行此示例给出：
pointless!
pointless!
pointless!
pointless!
this is bad
Traceback (most recent call last):
File "/home/john/projects/mpDemo.py", line 13, in mpFunctionReportError
rslt = f(**kwargs)
File "/home/john/projects/mpDemo.py", line 24, in raiseException
raise ValueError('this is bad')
ValueError: this is bad
this is bad
Traceback (most recent call last):
File "/home/john/projects/mpDemo.py", line 13, in mpFunctionReportError
rslt = f(**kwargs)
File "/home/john/projects/mpDemo.py", line 24, in raiseException
raise ValueError('this is bad')
ValueError: this is bad
答案 1 :(得分：1)
它不是很优雅，但是如何：
def func(args):
try:
# Emulate that the function might be fast or slow
time.sleep(random.randint(1,4))
ret = args["Np"] - args["index"]
# Emulate a bug
if args["index"]==args["problemIndex"]:
ret = 1/(args["index"]-args["problemIndex"])
# Return data
if args["runParallel"]:
# We use a queue thus ordering may not be protected
args["q"].put((args["index"],ret))
else:
return ret
except Exception as e:
logging.exception(e)
raise
输出应该如下所示（对于problemIndex = 9）：
ERROR:root:integer division or modulo by zero
Traceback (most recent call last):
File "/home/rciorba/test.py", line 26, in func
ret = 1/(args["index"]-args["problemIndex"])
ZeroDivisionError: integer division or modulo by zero
Bummer - one of more worker threads broke down 10 9
答案 2 :(得分：0)
John Greenall给出了最好的解决方案并且已经支付了赏金。
原因是他的解决方案没有尝试/除了代码的中心部分，即radu.ciorba向我们展示的整个“func”。然而，这种方式也是可行的。
由于Johns解决方案不是我的问题100％，我将在我自己的代码中发布一个解决方案，我已经应用了Johns解决方案。再次归功于约翰，也归功于拉杜！
#!/usr/bin/python
# map.py solution
import time
import multiprocessing
import sys
import random
import logging
import traceback
# Toggle whether we run parallel or not
runParallel = True
# Problematic index - if less than Np we create an exception
problemIndex = 14
# Number of compute problems
Np = 10
def func(args):
# Emulate that the function might be fast or slow
time.sleep(random.randint(1,4))
ret = args["Np"] - args["index"]
# Emulate a bug
if args["index"]==args["problemIndex"]:
ret = 1/(args["index"]-args["problemIndex"])
# Return data
return (args["index"],ret)
def mpFunctionReportError(args):
rslt=None
q = args["q"]
rslt = {"index":args["index"],
"args":None,
"error":None,
"traceback":None}
try:
rslt["result"] = func(args)
q.put(rslt)
except Exception as e:
rslt["result"] = None
rslt["error"] = e
rslt["args"] = str(args)
rslt["traceback"] = traceback.format_exc(e)
q.put(rslt)
# Return queue used when running parallel
manager = multiprocessing.Manager()
q = manager.Queue()
# Build argument lists
argList = []
for i in range(Np):
args={}
args["index"] = i # index
args["Np"] = Np # Number of problems
args["q"] = q # return queue for parallel execution mode
args["problemIndex"] = problemIndex # if index == problemIndex then func will malfunction
args["runParallel"] = runParallel # should we run parallel
argList.append(args)
resultVector = [None]*Np
#should we run parallel
if runParallel:
# Run 10 processes in parallel
p = multiprocessing.Pool(processes=10)
ret = p.map_async(mpFunctionReportError, argList)
# Wait until error or done
ret.wait()
# Queue size
qLen = q.qsize()
p.close()
# List for the errors
bugList = {}
# Loop the queue
for i in range(qLen):
# Pop a value
returnVal = q.get()
# Check for the error code
if returnVal["error"] is not None:
bugList[returnVal["index"]] = returnVal
else:
resultVector[returnVal["index"]] = returnVal["result"]
# Print the list of errors
if bugList:
print "-"*70
print "Some parts of the parallel execution broke down. Error list:"
print "-"*70
for i in bugList:
print "Index :",bugList[i]["index"]
print "Error code :",bugList[i]["error"]
print "Traceback :",bugList[i]["traceback"]
print "Args :",bugList[i]["args"]
print "-"*70
sys.exit(0)
else:
for p in range(Np):
resultVector[i] = func(argList[p])
for i in range(Np):
print "Index", i, "gives",resultVector[i]
当“runParallel = True”和“problemIndex = 4”中断时，我们现在有完整的跟踪信息
----------------------------------------------------------------------
Some parts of the parallel execution broke down. Error list:
----------------------------------------------------------------------
Index : 4
Error code : integer division or modulo by zero
Traceback : Traceback (most recent call last):
File "fix3.py", line 44, in mpFunctionReportError
rslt["result"] = func(args)
File "fix3.py", line 26, in func
ret = 1/(args["index"]-args["problemIndex"])
ZeroDivisionError: integer division or modulo by zero
Args : {'Np': 10, 'index': 4, 'problemIndex': 4, 'q': <AutoProxy[Queue] object, typeid 'Queue' at 0xb708710c>, 'runParallel': True}