Question

如何从进程池执行器中捕获异常？我已经使用了result（）但是它减慢了进程，我认为还有其他方法吗？

   with concurrent.futures.ProcessPoolExecutor(50) as genrator:
                    try:

                    for fl in self.finallist:

                        print("FIKLENAME",fl)

                        futures=[genrator.submit(insta.load_instantel_ascii, fl,None,',')]
                        results = [f.result() for f in futures]

                except Exception as e:

                    print("EXCE", e)
                    print("FILENAME IS",fl)

Answer 1

使用concurrent.futures.as_completed获取一个迭代器，在完成时产生未来。如果您使用此API，那么应该获得最小的收益。

还构建一个可在 for循环之外使用的期货的迭代。

def submit_task(executor, args):
    print('FILENAME', args[1])
    executor.submit(*args)

with concurrent.futures.ProcessPoolExecutor(50) as executor:
    try:
        futures = [
            submit_task(executor, (insta.load_instantel_ascii, fl, None, ',')) 
            for fl in self.finallist
        ]
        futures_as_completed = concurrent.futures.as_completed(futures)
        results = [future.result() for future in futures_as_completed]
    except Exception as exc:
        print("EXCEPTION: ", exc)

修改

要在异常块中获取fl，例如print('EXCEPTION: ', exc.fl)，你可以在调用者中捕获异常并使用绑定到fl的名称重新引发它，或者使用绑定到fl的属性实现自定义异常。呼叫者。

对于前者，如果您无权修改，则装饰insta.load_instantel_ascii 。

def try_except_fl(method, *args, **kwargs): try: return method(*args, **kwargs) except Exception as exc: fl = args[0] exc.fl = fl raise exc

如果您有权修改insta.load_instantel_ascii ，请将程序包装在try/except。

def load_instantel_ascii(self, fl, *args, **kwargs): try: # do some compute intensive tasks here except Exception as exc: exc.fl = fl raise exc

对于后者，使用绑定到fl的属性引发自定义异常。 e.g。

class ProcessFailure(Exception): # you may want to subclass a specialized exception class. message = 'Alas! This task had a lot of errors.' def __init__(fl): self.fl = fl

在失败点提高它。即 load_instantel_ascii。

def load_instantel_ascii(self, fl, *args, **kwargs): try: # do some compute intensive tasks here except Exception as exc: raise ProcessFailure(fl)

如何从python3中的processpool执行器中捕获异常

1 个答案: