Question

我有一个看起来像这样的脚本：

#!/usr/bin/env python
# encoding: utf-8

import time, random, os, multiprocessing

def main():
    NPROCESSES = 5
    pool = multiprocessing.Pool(processes=NPROCESSES)

    a = [1,2,3,4,5,6,7,8,9,0]
    for _ in pool.imap_unordered(do_task, a):
        pass

def do_task(n):
    try:
        might_crash(n)
    except Hell, e:
        print e, " crashed."

def might_crash(n):
    time.sleep(3*random.random())
    if random.randrange( 3 ) == 0:
        raise Hell(n)
    print n

class Hell(Exception):
    pass  

if __name__=="__main__":    
    main()

此脚本通常会打印“a”中的值，但是might_crash（）会随机引发异常。

我想捕获这些异常并将当前的do_task（）放回队列中以便稍后重试。

如果当前任务失败，如何将当前任务放回队列？

Answer 1

您可以从do_task收集结果，检查哪些结果是Hell的实例，将这些任务填充到列表new_tasks中，然后循环直到没有new_tasks：

import time
import random
import os
import multiprocessing as mp

def main():
    NPROCESSES = 5
    pool=mp.Pool(NPROCESSES)
    a = [1,2,3,4,5,6,7,8,9,0]
    new_tasks=a
    while new_tasks:
        a=new_tasks
        new_tasks=[]
        for result in pool.imap_unordered(do_task, a):
            if isinstance(result,Hell):
                new_tasks.append(result.args[0])
            else:
                print(result)

def do_task(n):
    try:
        result=might_crash(n)
    except Hell as e:        
        print("{0} crashed.".format(e.args[0]))
        result=e
    return result

def might_crash(n):
    time.sleep(3*random.random())
    if random.randrange( 3 ) == 0:
        raise Hell(n)
    return '{0} done'.format(n)

class Hell(Exception):
    pass  

if __name__=="__main__":    
    main()

产量

1 done
6 crashed.
4 done
7 crashed.
5 done
9 done
3 done
2 crashed.
8 done
0 crashed.
0 crashed.
2 done
7 crashed.
6 done
0 done
7 done

如果任务失败，如何将任务放回队列？

1 个答案: