我的问题与Combining asyncio with a multi-worker ProcessPoolExecutor非常相似-但是略有变化(我相信是async for
)使那里的出色答案对我来说毫无用处。
我正在尝试以下MWE:
import concurrent.futures
import asyncio
import time
async def mygen(u: int = 2):
i = 0
while i < u:
yield i
i += 1
def blocking(delay):
time.sleep(delay+1)
return('EXECUTOR: Completed blocking task number ' + str(delay+1))
async def non_blocking(loop):
with concurrent.futures.ProcessPoolExecutor() as executor:
async for i in mygen():
print('MASTER: Sending to executor blocking task number ' + str(i+1))
result = await loop.run_in_executor(executor, blocking, i)
print(result)
print('MASTER: Well done executor - you seem to have completed blocking task number ' + str(i+1))
loop = asyncio.get_event_loop()
loop.run_until_complete(non_blocking(loop))
按预期,此输出是非异步的:
MASTER: Sending to executor blocking task number 1
EXECUTOR: Completed blocking task number 1
MASTER: Well done executor - you seem to have completed blocking task number 1
MASTER: Sending to executor blocking task number 2
EXECUTOR: Completed blocking task number 2
MASTER: Well done executor - you seem to have completed blocking task number 2
我想调整代码,以使任务在两个并发进程中运行,并在输出可用时打印输出。所需的输出是:
MASTER: Sending to executor blocking task number 1
MASTER: Sending to executor blocking task number 2
EXECUTOR: Completed blocking task number 1
MASTER: Well done executor - you seem to have completed blocking task number 1
EXECUTOR: Completed blocking task number 2
MASTER: Well done executor - you seem to have completed blocking task number 2
从Combining asyncio with a multi-worker ProcessPoolExecutor我了解到,就目前情况而言,我await loop.run_in_executor()
的语法正在阻塞。我不知道如何以一种允许async for
移至下一个生成的值,同时等待执行者完成其工作的方式来替换它。请注意,我没有在他们的示例中使用asyncio.gather
。
答案 0 :(得分:0)
如果您最多要有两个进程来运行任务,那么最简单的实现方法是使用max_workers=2
创建执行程序。然后,您可以尽快提交任务,即继续进行async for
的下一个迭代,而不必等待上一个任务完成。您可以在最后收集所有任务的结果,以确保不会引起异常的注意(并可能获得实际结果)。
以下代码产生预期的输出:
from concurrent.futures import ProcessPoolExecutor
import asyncio
import time
async def mygen(u: int = 2):
i = 0
while i < u:
yield i
i += 1
def blocking(delay):
time.sleep(delay+1)
return('EXECUTOR: Completed blocking task number ' + str(delay+1))
async def run_blocking(executor, task_no, delay):
print('MASTER: Sending to executor blocking task number '
+ str(task_no))
result = await loop.run_in_executor(executor, blocking, delay)
print(result)
print('MASTER: Well done executor - you seem to have completed '
'blocking task number ' + str(task_no))
async def non_blocking(loop):
tasks = []
with ProcessPoolExecutor(max_workers=2) as executor:
async for i in mygen():
# spawn the task and let it run in the background
tasks.append(asyncio.create_task(
run_blocking(executor, i + 1, i)))
# if there was an exception, retrieve it now
await asyncio.gather(*tasks)
loop = asyncio.get_event_loop()
loop.run_until_complete(non_blocking(loop))