使用队列会导致asyncio异常“将未来<future未决=“”>附加到另一个循环中”

Question

我正在尝试使用asyncio队列运行此简单代码，但是捕获异常，甚至嵌套异常。

我想获得一些帮助使asyncio中的队列正常工作：

import asyncio, logging

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("asyncio").setLevel(logging.WARNING)


num_workers = 1
in_queue = asyncio.Queue()
out_queue = asyncio.Queue()
tasks = []


async def run():
    for request in range(1):
        await in_queue.put(request)

    # each task consumes from 'input_queue' and produces to 'output_queue':
    for i in range(num_workers):
        tasks.append(asyncio.create_task(worker(name=f'worker-{i}')))
    # tasks.append(asyncio.create_task(saver()))

    print('waiting for queues...')
    await in_queue.join()
    # await out_queue.join()
    print('all queues done')

    for task in tasks:
        task.cancel()
    print('waiting until all tasks cancelled')
    await asyncio.gather(*tasks, return_exceptions=True)
    print('done')


async def worker(name):
    while True:
        try:
            print(f"{name} started")
            num = await in_queue.get()
            print(f'{name} got {num}')
            await asyncio.sleep(0)
            # await out_queue.put(num)
        except Exception as e:
            print(f"{name} exception {e}")
        finally:
            print(f"{name} ended")
            in_queue.task_done()


async def saver():
    while True:
        try:
            print("saver started")
            num = await out_queue.get()
            print(f'saver got {num}')
            await asyncio.sleep(0)
            print("saver ended")
        except Exception as e:
            print(f"saver exception {e}")
        finally:
            out_queue.task_done()


asyncio.run(run(), debug=True)
print('Done!')

输出：

waiting for queues...
worker-0 started
worker-0 got 0
worker-0 ended
worker-0 started
worker-0 exception 
worker-0 ended
ERROR:asyncio:unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<worker() done, defined at temp4.py:34> exception=ValueError('task_done() called too many times') created at Python37\lib\asyncio\tasks.py:325>
Traceback (most recent call last):
  File "Python37\lib\asyncio\runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "Python37\lib\asyncio\base_events.py", line 573, in run_until_complete
    return future.result()
  File "temp4.py", line 23, in run
    await in_queue.join()
  File "Python37\lib\asyncio\queues.py", line 216, in join
    await self._finished.wait()
  File "Python37\lib\asyncio\locks.py", line 293, in wait
    await fut
RuntimeError: Task <Task pending coro=<run() running at temp4.py:23> cb=[_run_until_complete_cb() at Python37\lib\asyncio\base_events.py:158] created at Python37\lib\asyncio\base_events.py:552> got Future <Future pending> attached to a different loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "temp4.py", line 46, in worker
    in_queue.task_done()
  File "Python37\lib\asyncio\queues.py", line 202, in task_done
    raise ValueError('task_done() called too many times')
ValueError: task_done() called too many times
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\pydevd.py", line 1664, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "temp4.py", line 63, in <module>
    asyncio.run(run(), debug=True)
  File "Python37\lib\asyncio\runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "Python37\lib\asyncio\base_events.py", line 573, in run_until_complete
    return future.result()
  File "temp4.py", line 23, in run
    await in_queue.join()
  File "Python37\lib\asyncio\queues.py", line 216, in join
    await self._finished.wait()
  File "Python37\lib\asyncio\locks.py", line 293, in wait
    await fut
RuntimeError: Task <Task pending coro=<run() running at temp4.py:23> cb=[_run_until_complete_cb() at Python37\lib\asyncio\base_events.py:158] created at Python37\lib\asyncio\base_events.py:552> got Future <Future pending> attached to a different loop

这是基本流程，我稍后要执行的操作是在更多工人上运行更多请求，其中每个工人会将数字从in_queue移到out_queue，然后保护程序将打印数字来自out_queue。

Answer 1

必须在循环内创建您的队列。您是在为asyncio.run()创建的循环之外创建它们的，因此它们使用events.get_event_loop()。 asyncio.run()创建了一个新循环，并且在一个循环中为队列创建的期货无法在另一个循环中使用。

在顶级run()协程中创建队列，然后将其传递给需要它们的协程，或者如果必须使用全局变量，则使用contextvars.ContextVar objects。

您还需要清理如何处理任务中的任务取消。通过在任务中举起asyncio.CancelledError exception 可以取消任务。您可以忽略它，但是如果抓住它进行清理工作，您必须重新提出它。

您的任务代码无需重新引发即可捕获所有异常，包括CancelledError，因此您可以阻止适当的取消。

相反，取消期间发生的 是您致电queue.task_done()；不要这样做，至少在取消任务时不要这样做。您仅应在实际处理队列任务时调用task_done()，而在等待队列任务出现时在发生异常时代码调用task_done() 。

如果您需要使用try...finally: in_queue.task_done()，请将其放在处理从队列中接收到的项目的代码块中，并将await in_queue.get()保留在{{ 1}}块。您不想标记您实际上没有收到的任务。

最后，当您打印例外时，您要打印其try；出于历史原因，repr()异常的转换会产生其str()的值，这对于.args异常（其值为空的CancelledError）不是很有用。在格式化的字符串中使用.args，这样您就可以看到正在捕获的异常：

{e!r}

因此，在启用了worker-0 exception CancelledError() 任务，在saver()内部创建的队列以及清理了任务异常处理的情况下，更正后的代码将是：

run()

此打印

import asyncio, logging

logging.basicConfig(level=logging.DEBUG)
logging.getLogger("asyncio").setLevel(logging.WARNING)


num_workers = 1


async def run():
    in_queue = asyncio.Queue()
    out_queue = asyncio.Queue()

    for request in range(1):
        await in_queue.put(request)

    # each task consumes from 'in_queue' and produces to 'out_queue':
    tasks = []
    for i in range(num_workers):
        tasks.append(asyncio.create_task(
            worker(in_queue, out_queue, name=f'worker-{i}')))
    tasks.append(asyncio.create_task(saver(out_queue)))

    await in_queue.join()
    await out_queue.join()

    for task in tasks:
        task.cancel()

    await asyncio.gather(*tasks, return_exceptions=True)

    print('done')

async def worker(in_queue, out_queue, name):
    print(f"{name} started")
    try:
        while True:
            num = await in_queue.get()
            try:
                print(f'{name} got {num}')
                await asyncio.sleep(0)
                await out_queue.put(num)
            except Exception as e:
                print(f"{name} exception {e!r}")
                raise
            finally:
                in_queue.task_done()
    except asyncio.CancelledError:
        print(f"{name} is being cancelled")
        raise
    finally:
        print(f"{name} ended")

async def saver(out_queue):
    print("saver started")
    try:
        while True:
            num = await out_queue.get()
            try:
                print(f'saver got {num}')
                await asyncio.sleep(0)
                print("saver ended")
            except Exception as e:
                print(f"saver exception {e!r}")
                raise
            finally:
                out_queue.task_done()
    except asyncio.CancelledError:
        print(f"saver is being cancelled")
        raise
    finally:
        print(f"saver ended")

asyncio.run(run(), debug=True)
print('Done!')

如果要使用全局变量来共享队列对象，请使用worker-0 started worker-0 got 0 saver started saver got 0 saver ended done worker-0 is being cancelled worker-0 ended saver is being cancelled saver ended Done! 对象。您仍然可以在ContextVar中创建队列，但是如果要启动多个循环，则run()模块集成将负责将队列分开：

contextvars

Answer 2

当将队列作为参数传递不是一个选项时，您也可以使用预先创建的事件循环显式初始化它

loop = asyncio.get_event_loop()
queue = asyncio.Queue(loop=loop)

然而，在这种情况下，您将放弃 asyncio.run 方法的实用程序，而必须自己处理事件循环的启动和关闭

try:
    asyncio.set_event_loop(loop)
    loop.set_debug(True)
    loop.run_until_complete(run())
finally:
    try:
        asyncio.runners._cancel_all_tasks(loop)
        loop.run_until_complete(loop.shutdown_asyncgens())
    finally:
        asyncio.set_event_loop(None)
        loop.close()