如何用电机彻底关闭Change Streams?

时间:2018-01-05 12:54:02

标签: python mongodb python-3.x python-asyncio pymongo-3.x

TL; DR

这确实是Motor 1.2.0中的一个错误,该错误由A. Jesse Jiryu Davis迅速修复,并且可以在1.2.1或更高版本的驱动程序中使用。

原始问题

我在Python 3上编写了一个程序,用于监视MongoDB集合的更改,使用其新的Change Stream功能。这是MCVE:

from asyncio import get_event_loop, CancelledError
from contextlib import suppress
from motor.motor_asyncio import AsyncIOMotorClient

async def watch(collection):
    async with collection.watch([]) as stream:
        async for change in stream:
            print(change)

async def cleanup():
    task.cancel()

    with suppress(CancelledError):
        await task

if __name__ == '__main__':
    conn = AsyncIOMotorClient()
    loop = get_event_loop()
    task = loop.create_task(watch(conn.database.collection))  # Replace with a real collection.

    try:
        loop.run_forever()

    except KeyboardInterrupt:
        pass

    finally:
        loop.run_until_complete(cleanup())
        loop.shutdown_asyncgens()
        loop.close()

当我用CRTL + C杀死程序时,它会引发三种不同的例外。

^Cexception calling callback for <Future at 0x102efea58 state=finished raised InvalidStateError>
Traceback (most recent call last):
  File "/Users/viotti/motor/lib/python3.6/site-packages/motor/core.py", line 1259, in _next
    change = self.delegate.next()
  File "/Users/viotti/motor/lib/python3.6/site-packages/pymongo/change_stream.py", line 79, in next
    change = self._cursor.next()
  File "/Users/viotti/motor/lib/python3.6/site-packages/pymongo/command_cursor.py", line 292, in next
    raise StopIteration
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/viotti/motor/lib/python3.6/site-packages/motor/core.py", line 1264, in _next
    future.set_exception(StopAsyncIteration())
asyncio.base_futures.InvalidStateError: invalid state

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 324, in _invoke_callbacks
    callback(self)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/futures.py", line 414, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 620, in call_soon_threadsafe
    self._check_closed()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

有没有办法让这个程序默默关闭?

我在macOS Sierra上使用Python 3.6.4,Motor 1.2和pymongo 3.6.0进行测试。

1 个答案:

答案 0 :(得分:2)

我认为您的代码是正确的,motor方面存在问题。

在调查时我发现了两个问题:

  1. 如果当前没有建立闭环连接,那么在异步回调完成之前,循环关闭后会出现exception calling callback for <Future错误。它似乎与异步生成器或流无关,而与任何motor用法有关。
  2. AgnosticChangeStream异步迭代机制(_next function)在没有考虑取消的情况下编写。尝试将取消的未来设置为InvalidStateError
  3. 此代码演示了两个问题和可能的解决方法:

    import types
    import asyncio
    from contextlib import suppress
    from motor.motor_asyncio import AsyncIOMotorClient
    
    
    async def test():
        while True:
            await asyncio.sleep(0.1)
    
    
    async def cleanup(task):
        task.cancel()
        with suppress(asyncio.CancelledError):
            await task
    
    
    def _next(self, future):
        try:
            if not self.delegate:
                self.delegate = self._collection.delegate.watch(**self._kwargs)
    
            change = self.delegate.next()
            self._framework.call_soon(self.get_io_loop(),
                                      future.set_result,
                                      change)
        except StopIteration:
            future.set_exception(StopAsyncIteration())
        except Exception as exc:
    
            # CASE 2:
            # Cancellation of async iteration (and future with it) happens immediately
            # and trying to set exception to cancelled future leads to InvalidStateError,
            # we should prevent it:
            if future.cancelled():
                return
    
            future.set_exception(exc)
    
    
    async def watch(collection):
        async with collection.watch([]) as stream:
    
            # Patch stream to achieve CASE 2:
            stream._next = types.MethodType(_next, stream)
    
            async for change in stream:
                print(change)
    
    
    if __name__ == '__main__':
        loop = asyncio.get_event_loop()
        tmp = asyncio.ensure_future(test())  # Way to receive KeyboardInterrupt immediately.
    
        client = AsyncIOMotorClient()
        collection = client.test_database.test_collection
        task = asyncio.ensure_future(watch(collection))
    
        try:
            loop.run_forever()
        except KeyboardInterrupt:
            print('KeyboardInterrupt')
        finally:
            loop.run_until_complete(cleanup(tmp))
            loop.run_until_complete(cleanup(task))
    
            # CASE 1:
            # Looks like propagating KeyboardInterrupt doesn't affect motor's try
            # to establish connection to db and I didn't find a way to stop this manually.
            # We should keep event loop alive until we receive ServerSelectionTimeoutError
            # and motor would be able to execute it's asyncio callbacks:
            loop.run_until_complete(asyncio.sleep(client.server_selection_timeout))
    
            loop.shutdown_asyncgens()
            loop.close()
    

    由于添加了修复,它完成时没有警告/异常(至少在我的机器上)。

    我不建议你使用上面的黑客!这只是为了展示问题所在和可能的解决方案。我不确定它能做好一切。

    相反,我建议你create issue at motor user group / Jira附加你的片段,可能是我的答案,等到bug修好。