我试图以这种方式使用python的多处理包:
featureClass = [[1000,k,1] for k in drange(start,end,step)] #list of arguments
for f in featureClass:
pool .apply_async(worker, args=f,callback=collectMyResult)
pool.close()
pool.join
从池的进程中我想避免等待超过60秒的那些返回其结果。这可能吗?
答案 0 :(得分:25)
这是您无需更改worker
功能即可完成此操作的方法。我们的想法是将worker包装在另一个函数中,该函数将在后台线程中调用worker
,然后等待timeout
秒的结果。如果超时到期,则会引发异常,这将突然终止正在执行的线程worker
:
import multiprocessing
from multiprocessing.dummy import Pool as ThreadPool
from functools import partial
def worker(x, y, z):
pass # Do whatever here
def collectMyResult(result):
print("Got result {}".format(result))
def abortable_worker(func, *args, **kwargs):
timeout = kwargs.get('timeout', None)
p = ThreadPool(1)
res = p.apply_async(func, args=args)
try:
out = res.get(timeout) # Wait timeout seconds for func to complete.
return out
except multiprocessing.TimeoutError:
print("Aborting due to timeout")
p.terminate()
raise
if __name__ == "__main__":
pool = multiprocessing.Pool()
featureClass = [[1000,k,1] for k in drange(start,end,step)] #list of arguments
for f in featureClass:
abortable_func = partial(abortable_worker, worker, timeout=3)
pool.apply_async(abortable_func, args=f,callback=collectMyResult)
pool.close()
pool.join()
超时的所有函数都会引发multiprocessing.TimeoutError
。请注意,这意味着当超时发生时,您的回调不会执行。如果这不可接受,只需更改except
的{{1}}块即可返回内容,而不是调用abortable_worker
。
答案 1 :(得分:2)
我们可以使用gevent.Timeout来设置工作人员的运行时间。 gevent tutorial
from multiprocessing.dummy import Pool
#you should install gevent.
from gevent import Timeout
from gevent import monkey
monkey.patch_all()
import time
def worker(sleep_time):
try:
seconds = 5 # max time the worker may run
timeout = Timeout(seconds)
timeout.start()
time.sleep(sleep_time)
print "%s is a early bird"%sleep_time
except:
print "%s is late(time out)"%sleep_time
pool = Pool(4)
pool.map(worker, range(10))
output:
0 is a early bird
1 is a early bird
2 is a early bird
3 is a early bird
4 is a early bird
8 is late(time out)
5 is late(time out)
6 is late(time out)
7 is late(time out)
9 is late(time out)