Question

我试图以这种方式使用python的多处理包：

featureClass = [[1000,k,1] for k in drange(start,end,step)] #list of arguments
for f in featureClass:
  pool .apply_async(worker, args=f,callback=collectMyResult)
pool.close()
pool.join

从池的进程中我想避免等待超过60秒的那些返回其结果。这可能吗？

Answer 1

这是您无需更改worker功能即可完成此操作的方法。我们的想法是将worker包装在另一个函数中，该函数将在后台线程中调用worker，然后等待timeout秒的结果。如果超时到期，则会引发异常，这将突然终止正在执行的线程worker：

import multiprocessing
from multiprocessing.dummy import Pool as ThreadPool
from functools import partial

def worker(x, y, z):
    pass # Do whatever here

def collectMyResult(result):
    print("Got result {}".format(result))

def abortable_worker(func, *args, **kwargs):
    timeout = kwargs.get('timeout', None)
    p = ThreadPool(1)
    res = p.apply_async(func, args=args)
    try:
        out = res.get(timeout)  # Wait timeout seconds for func to complete.
        return out
    except multiprocessing.TimeoutError:
        print("Aborting due to timeout")
        p.terminate()
        raise

if __name__ == "__main__":
    pool = multiprocessing.Pool()
    featureClass = [[1000,k,1] for k in drange(start,end,step)] #list of arguments
    for f in featureClass:
      abortable_func = partial(abortable_worker, worker, timeout=3)
      pool.apply_async(abortable_func, args=f,callback=collectMyResult)
    pool.close()
    pool.join()

超时的所有函数都会引发multiprocessing.TimeoutError。请注意，这意味着当超时发生时，您的回调不会执行。如果这不可接受，只需更改except的{{1}}块即可返回内容，而不是调用abortable_worker。

Answer 2

我们可以使用gevent.Timeout来设置工作人员的运行时间。 gevent tutorial

from multiprocessing.dummy import Pool 
#you should install gevent.
from gevent import Timeout
from gevent import monkey
monkey.patch_all()
import time

def worker(sleep_time):
    try:

        seconds = 5  # max time the worker may run
        timeout = Timeout(seconds) 
        timeout.start()
        time.sleep(sleep_time)
        print "%s is a early bird"%sleep_time
    except:
        print "%s is late(time out)"%sleep_time

pool = Pool(4)

pool.map(worker, range(10))


output:
0 is a early bird
1 is a early bird
2 is a early bird
3 is a early bird
4 is a early bird
8 is late(time out)
5 is late(time out)
6 is late(time out)
7 is late(time out)
9 is late(time out)

如何在超时后中止multiprocessing.Pool中的任务？

2 个答案: