Question

我有一大堆要插入MongoDB的文档（可能n> 100000）。我不想一次创建100000个延迟，但我不想执行并顺序等待每个查询，因为我有一个连接池到MongoDB，我想完全利用它。所以我有一个生成器函数，它将产生DeferredLazyList消耗的延迟。

def generate_update_deferreds(collection, many_docs):
    for doc in many_docs:
        d = collection.update({'_id': doc['_id']}, doc, upsert=True)
        yield d

这是连接延迟upsert的生成和DeferredLazyList的代码。

@defer.inlineCallbacks
def update_docs(collection, many_docs):
    gen_deferreds = generate_update_deferreds(collection, many_docs)
    results = yield DeferredLazyList(gen_deferreds, count=pool_size, consume_errors=True)

DeferredLazyList类似于DeferredList，但不是接受延迟列表来等待它接受迭代器。延迟从迭代器中检索，同时只有count延迟激活。这用于有效地批量延迟，因为它们是在产生时创建的。

class DeferredLazyList(defer.Deferred):
    """
    The ``DeferredLazyList`` class is used for collecting the results of
    many deferreds. This is similar to ``DeferredList``
    (``twisted.internet.defer.DeferredList``) but works with an iterator
    yielding deferreds. This will only maintain a certain number of
    deferreds simultaneously. Once one of the deferreds finishes, another
    will be obtained from the iterator.
    """

    def __init__(self, deferreds, count=None, consume_errors=None):
        defer.Deferred.__init__(self)

        if count is None:
            count = 1

        self.__consume_errors = bool(consume_errors)

        self.__iter = enumerate(deferreds)
        self.__results = []
        for _i in xrange(count):
            # Start specified number of simultaneous deferreds.
            if not self.called:
                self.__next_save_result(None, None, None)
            else:
                break

    def __next_save_result(self, result, success, index):
        """
        Called when a deferred completes.
        """
        # Make sure we can save result at index.
        if index is not None:
            results_len = len(self.__results)
            if results_len <= index:
                self.__results += [NO_RESULT] * (index - results_len + 1)
            # Save result.
            self.__results[index] = (success, result)

        # Get next deferred.
        try:
            i, d = self.__iter.next()
            d.addCallbacks(self.__next_save_result, self.__next_save_result, callbackArgs=(True, i), errbackArgs=(False, i))

        except StopIteration:
            # Iterator is exhausted, callback self with results.
            self.callback(self.__results)

        # Pass through result.
        return result if success or not self.__consume_errors else None

问题在于，generate_update_deferreds()已将.called True设置为DeferredLazyList，导致DeferredLazyList.__init__()以递归方式自行调用。

发生的事情是：

在self.__next_save_result()中，count被称为self.__next_save_result()次（比如5）。
对self.__iter的每次调用都会消耗1 .called延迟，并且本身会被添加为回调。
由于屈服延迟已将True设置为d.addCallbacks(self.__next_save_result, ...)，self.__next_save_result()会立即调用RuntimeError，此循环将继续，直到File "/home/caleb/it/Development/projects/python/amazon/bin/feeds-daemon/lib/server.py", line 937, in update_many_docs results = yield DeferredLazyList(gen_deferreds, count=self.mongo_connections, consume_errors=True, return_results=True) File "/home/caleb/it/Development/projects/python/amazon/bin/feeds-daemon/lib/twisted.py", line 157, in __init__ self.__next_save_result(None, None, None) File "/home/caleb/it/Development/projects/python/amazon/bin/feeds-daemon/lib/twisted.py", line 222, in __next_save_result d.addCallbacks(self.__next_save_result, self.__next_save_result, callbackArgs=(True, i), errbackArgs=(False, i)) File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 290, in addCallbacks self._runCallbacks() File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 551, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/caleb/it/Development/projects/python/amazon/bin/feeds-daemon/lib/twisted.py", line 222, in __next_save_result d.addCallbacks(self.__next_save_result, self.__next_save_result, callbackArgs=(True, i), errbackArgs=(False, i)) File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 290, in addCallbacks self._runCallbacks() File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 551, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/caleb/it/Development/projects/python/amazon/bin/feeds-daemon/lib/twisted.py", line 222, in __next_save_result d.addCallbacks(self.__next_save_result, self.__next_save_result, callbackArgs=(True, i), errbackArgs=(False, i)) # Repeated until the RuntimeError exceptions.RuntimeError: maximum recursion depth exceeded被提升为止已达到递归深度。

我在达到递归限制之前打印了一个堆栈跟踪，以确认这是导致问题的原因：

cooperate()

非常感谢任何帮助。顺便说一下，我正在使用Twisted 12.1.0运行Python 2.7.3，并且MongoDB的内容实际上只与理解上下文有关。

我想要每个延迟的结果，但是CooperativeTask不返回那些，所以我在每个延迟之前添加了一个回调，然后将它们交给from twisted.internet.defer import DeferredList, inlineCallbacks from twisted.internet.task import cooperate NO_RESULT = object() def generate_update_deferreds(collection, many_docs, save_results): for i, doc in enumerate(update_docs): d = collection.update({'_id': doc['_id']}, doc, upsert=True) d.addBoth(save_result, i, save_results) # Save result yield d def save_result(result, i, save_results): save_results[i] = result @inlineCallbacks def update_docs(collection, many_docs): save_results = [NO_RESULT] * len(many_docs) gen_deferreds = generate_update_deferreds(collection, many_docs, save_results)) workers = [cooperate(gen_deferreds).whenDone() for _i in xrange(count)] yield defer.DeferredList(workers) # Handle save_results...：

{{1}}

Answer 1

Twisted中有一些工具可以帮助您更轻松地完成此任务。例如，合作：

from twisted.internet.task import cooperate

def generate_update_deferreds(collection, many_docs):
    for doc in update_docs:
        d = collection.update({'_id': doc['_id']}, doc, upsert=True)
        yield d

work = generate_update_deferreds(...)
worker_tasks = []
for i in range(count):
    task = cooperate(work)
    worker_tasks.append(task)

all_done_deferred = DeferredList([task.whenDone() for task in worker_tasks])

延迟延迟列表达到最大递归深度

1 个答案: