我有一个long_task
函数,它运行繁重的cpu绑定计算,我希望通过使用新的asyncio框架使其异步。生成的long_task_async
函数使用ProcessPoolExecutor
将工作卸载到不同的进程,不受GIL约束。
问题在于,由于某种原因,concurrent.futures.Future
实例从ProcessPoolExecutor.submit
返回时从投掷TypeError
投掷。这是设计的吗?这些期货是否与asyncio.Future
类不兼容?什么是解决方法?
我还注意到生成器不可拾取,因此向ProcessPoolExecutor
提交couroutine将失败。对此还有什么干净的解决方案吗?
import asyncio
from concurrent.futures import ProcessPoolExecutor
@asyncio.coroutine
def long_task():
yield from asyncio.sleep(4)
return "completed"
@asyncio.coroutine
def long_task_async():
with ProcessPoolExecutor(1) as ex:
return (yield from ex.submit(long_task)) #TypeError: 'Future' object is not iterable
# long_task is a generator, can't be pickled
loop = asyncio.get_event_loop()
@asyncio.coroutine
def main():
n = yield from long_task_async()
print( n )
loop.run_until_complete(main())
答案 0 :(得分:10)
您想使用loop.run_in_executor
,它使用concurrent.futures
执行程序,但会将返回值映射到asyncio
的未来。
原始asyncio
PEP suggests that concurrent.futures.Future
有朝一日可能会增加__iter__
方法,因此它也可以与yield from
一起使用,但目前图书馆被设计为仅需要yield from
支持,仅此而已。 (否则一些代码在3.3中实际上不起作用。)
答案 1 :(得分:10)
我们可以通过调用concurrent.futures.Future
将asyncio.future
打包到asyncio.wrap_future(Future)
。我用下面的代码试了一下。工作正常
from asyncio import coroutine
import asyncio
from concurrent import futures
def do_something():
ls = []
for i in range(1, 1000000):
if i % 133333 == 0:
ls.append(i)
return ls
@coroutine
def method():
with futures.ProcessPoolExecutor(max_workers=10) as executor:
job = executor.submit(do_something)
return (yield from asyncio.wrap_future(job))
@coroutine
def call_method():
result = yield from method()
print(result)
def main():
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(call_method())
finally:
loop.close()
if __name__ == '__main__':
main()