Question

使用python多处理池时，是否可以在准备好后获得结果？

例如类似这样的地方，results在准备就绪时产生结果，而不仅仅是所有结果都准备就绪。

args = [ ... ]
def foo():
    pass

pool = multiprocessing.Pool()
results = pool.map_async(foo, l)

for result in results:
    save_result(result)

我能想到的唯一方法是

args = [ ... ]
results = []
def foo():
    pass

pool = multiprocessing.Pool()

for arg in args:
    results.append(pool.apply_async(foo, arg))

while results:
    for result in results:
        if result.ready():
            save_result(result.get())

我在Windows 7 x64上使用Python 3.4

Answer 1

由于您使用的是python 3.2+，最简单的方法是使用concurrent.futures代替multiprocessing.Pool。 concurrent.futures.as_completed将为您提供一个将产生结果的迭代器..好了，完成后。

基本用法结构：

with concurrent.futures.ProcessPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(foo, arg) for arg in some_args]
    for future in concurrent.futures.as_completed(futures):
        #do something with completed result

请参阅链接的文档，它还有一些示例可帮助您入门。

Answer 2

其他响应建议使用parallel.futures，但是对于某些应用程序，进程不是线程安全的，而是需要使用多处理来完成。对于此应用程序，我找到了一种解决您所要问题的方法。

pool = multiprocessing.Pool()
pool_results = []
pool_results.append(pool.apply_async(<function>, (<args>)))
# ... add more function calls. possibly using loop ...

pool.close()
while len(pool_results) > 0:
    to_remove = [] #avoid removing objects during for_loop
    for r in pool_results:
        # check if process is finished
        if r.ready(): 
            # print result (or do any operation with result)
            print(r.get())
            to_remove.append(r)
    for remove in to_remove:
        pool_results.remove(remove)
    time.sleep(1) # ensures that this thread doesn't consume too much memory
pool.join() # make sure all processes are completed

是否可以在完成后获得python池结果？

2 个答案: