Question

我正在使用pool.imap_unordered对本地保存的不同txt文件应用功能。

是否可以捕获异常和pass？如果我的代码遇到异常，则会阻塞整个循环。

pool = Pool(processes=15)

results = {}
files = glob.glob('{}/10K_files/*.txt'.format(path_input))

for key, output in tqdm(pool.imap_unordered(process_file, files),total=len(files)):
    results[key] = output

我尝试过这样的事情：

pool = Pool(processes=15)

results = {}
files = glob.glob('{}/10K_files/*.txt'.format(path_input))

try:
    for key, output in tqdm(pool.imap_unordered(process_file, files), total=len(files)):
       results[key] = output
except:
    print("error")

但是我想从我开始的地方继续循环。

谢谢！

Answer 1

您可以在process_file中捕获异常并将其返回。然后测试返回值是否为异常。这是一个示例：

import os
import traceback
import multiprocessing as mp


def main():
    work_items = [i for i in range(20)]
    pool = mp.Pool()
    for result in pool.imap_unordered(process_file_exc, work_items):
        if isinstance(result, Exception):
            print("Got exception: {}".format(result))
        else:
            print("Got OK result: {}".format(result))


def process_file_exc(work_item):
    try:
        return process_file(work_item)
    except Exception as ex:
        return Exception("Err on item {}".format(work_item)
                         + os.linesep + traceback.format_exc())


def process_file(work_item):
    if work_item == 9:
        # this will raise ZeroDivisionError exception
        return work_item / 0
    return "{} * 2 == {}".format(work_item, work_item * 2)


if __name__ == '__main__':
    main()

如何在python多处理中使用imap_unordered处理异常

1 个答案: