Question

我试图以异步方式在请求之间添加延迟。当我使用Tornado gen.sleep（x）时，我的函数（启动）不会被执行。如果我从 yield gen.sleep（1.0）中删除 yield ，则会调用函数，但不会添加任何延迟。如何在for循环中的请求之间添加延迟？我需要每秒控制外部API的请求。如果我使用 time.sleep ，则在完成所有请求后，响应会延迟。试图添加@ gen.engine装饰器来启动功能，没有结果。

代码：

import collections
import tornado.httpclient


class BacklogClient(object):
    MAX_CONCURRENT_REQUESTS = 20

    def __init__(self, ioloop):
        self.ioloop = ioloop
        self.client = tornado.httpclient.AsyncHTTPClient(max_clients=self.MAX_CONCURRENT_REQUESTS)
        self.client.configure(None, defaults=dict(connect_timeout=20, request_timeout=30))
        self.backlog = collections.deque()
        self.concurrent_requests = 0

    def __get_callback(self, function):
        def wrapped(*args, **kwargs):
            self.concurrent_requests -= 1
            self.try_run_request()
            return function(*args, **kwargs)

        return wrapped

    def try_run_request(self):
        while self.backlog and self.concurrent_requests < self.MAX_CONCURRENT_REQUESTS:
            request, callback = self.backlog.popleft()
            self.client.fetch(request, callback=callback)
            self.concurrent_requests += 1

    def fetch(self, request, callback=None):
        wrapped = self.__get_callback(callback)

        self.backlog.append((request, wrapped))
        self.try_run_request()


import time
from tornado import ioloop, httpclient, gen


class TornadoBacklog:
    def __init__(self):

        self.queue = 0
        self.debug = 1
        self.toProcess = [
            'http://google.com',
            'http://yahoo.com',
            'http://nytimes.com',
            'http://msn.com',
            'http://cnn.com',
            'http://twitter.com',
            'http://facebook.com',
        ]


    def handle_request(self, response):

        print response.code
        if not self.backlog.backlog and self.backlog.concurrent_requests == 0:
            ioloop.IOLoop.instance().stop()


    def launch(self):
        self.ioloop = ioloop.IOLoop.current()
        self.backlog = BacklogClient(self.ioloop)

        for item in self.toProcess:
            yield gen.sleep(1.0)
            print item
            self.backlog.fetch(
                httpclient.HTTPRequest(
                    item,
                    method='GET',
                    headers=None,
                ),
                self.handle_request
            )

        self.ioloop.start()



def main():
    start_time = time.time()

    scraper = TornadoBacklog()
    scraper.launch()

    elapsed_time = time.time() - start_time
    print('Process took %f seconds processed %d items.' % (elapsed_time, len(scraper.toProcess)))


if __name__ == "__main__":
    main()

参考：https://github.com/tornadoweb/tornado/issues/1400

Answer 1

Tornado协同程序有两个组成部分：

它们包含“收益”声明
用“gen.coroutine”

在“启动”功能中使用“coroutine”装饰器：

@gen.coroutine
def launch(self):

从头到尾运行Tornado协同程序，如下所示：

tornado.ioloop.IOLoop.current().run_sync(launch)

从“启动”功能中删除对“ioloop.start”的调用：循环运行“启动”功能，反之亦然。

Tornado gen.sleep添加延迟

1 个答案: