Python:完成任务排序后获取对原始任务的引用

时间:2018-04-25 17:49:15

标签: python-asyncio

问题:在asyncio.as_completed产生结果后,您如何获得对原始任务的引用?

除了Python之外,基本上和C#问题一样:Getting reference to original Task after ordering Tasks by completion?

示例问题:

# Takes a list of WebClient objects,
# calls each one simultaneously,
# and yields the results immediately as they arrive
# to a synchronous caller.

def yieldThingsAsTheyArrive(webClients):

    tasks = []
    for webClient in webClients:
        # This is what we want to get a reference to later:
        task = webClient.fetch_thing()  # start long-running request asynchronously
        tasks.append(task)

    loop = asyncio.get_event_loop()
    for future in asyncio.as_completed(tasks):
        thing = loop.run_until_complete(future)  # since our caller is synchronous, wait until the task completes so we can yield the final result instead of a future
        thing.originalWebClient = ???  # This is where we need a reference to the original webClient
        yield thing

1 个答案:

答案 0 :(得分:3)

as_completed具体是因为它既不会产生像asyncio.wait这样的未来,也不会产生像asyncio.gather这样的结果。相反,它会产生您需要等待的协程(以您喜欢的任何方式)以完成顺序获得结果。它不会产生你传递给它的期货,因为那时它还不知道下一个通过的期货将会完成。

您可以通过在另一个未来包装任务来关联任意数据,其结果是任务对象(您已将数据附加到该对象)。这基本上等同于C# code所做的,只是没有静态打字仪式。从this answer进行设置,可运行的示例如下所示:

import asyncio

async def first():
    await asyncio.sleep(5)
    return 'first'

async def second():
    await asyncio.sleep(1)
    return 'second'

async def third():
    await asyncio.sleep(3)
    return 'third'

def ordinary_generator():
    loop = asyncio.get_event_loop()

    wrappers = []
    for idx, coro in enumerate((first(), second(), third())):
        task = loop.create_task(coro)
        task.idx = idx + 1
        # Wrap the task in a future that completes when the 
        # task does, but whose result is the task object itself.
        wrapper = loop.create_future()
        task.add_done_callback(wrapper.set_result)
        wrappers.append(wrapper)

    for x in asyncio.as_completed(wrappers):
        # yield completed tasks
        yield loop.run_until_complete(x)

for task in ordinary_generator():
    print(task.result(), task.idx)

我建议的另一个选项是使用调用as_completed的循环替换asyncio.wait(return_when=FIRST_COMPLETED)上的迭代。这也将提供期货,因为它们是完整的,但不需要额外的包装,并导致稍微更惯用的asyncio代码。我们在每个协程上调用ensure_future将其转换为将来,将数据附加到其中,然后将其传递给asyncio.wait()。由于wait返回那些相同的未来,所附的数据就在它们上面。

def ordinary_generator():
    loop = asyncio.get_event_loop()

    pending = []
    for idx, coro in enumerate((first(), second(), third())):
        task = loop.create_task(coro)
        task.idx = idx + 1
        pending.append(task)

    while pending:
        done, pending = loop.run_until_complete(asyncio.wait(
            pending, return_when=asyncio.FIRST_COMPLETED))
        for task in done:
            yield task