Question

我正在使用python发送多个HTTP请求。＆gt;同时异步触发1000个请求。但我收到错误：达到max_clients限制，请求排队龙卷风，过了一会儿我收到超时错误。

如何解决此类问题，发送多个http post请求并避免超时？

以下是我正在使用的代码：

class AjaxBatchHandler(basehandler.BaseHandler):
    @tornado.gen.coroutine
    def post(self):
        # just respond with a status, no redirect to login
        if not self.get_current_user:
            self.set_status(403)

        batch = json.loads(self.get_argument("actions").encode('utf-8'))

        client = tornado.httpclient.AsyncHTTPClient()

        batch_requests = []
        for item in batch['actions']:
            request = utils.build_request(
                self,
                action=item['action'].replace("{API}", utils.get_api_baseurl()),
                values=item['values'])
            batch_requests.append(client.fetch(request))

        try:
            batch_responses = yield batch_requests

            batch_result = dict(results=[])
            for result in batch_responses:
                batch_result['results'].append(json.loads(result.body))

        except tornado.httpclient.HTTPError as e:
            batch_result = dict(results=[])
            batch_result['results'].append({"Status": 500,
                                        "StatusMsg": e.message,
                                        "Error": e.code
                                        })

        self.write(batch_result)

Answer 1

要么增加max_clients限制（如果适合您向您正在发送的网站发送更多流量），要么降低您的请求速度。对于前者，请

AsyncHTTPClient.configure(None, max_clients=1000)

在您的计划开始时。对于后者，信号量或队列可用于控制发送请求的速率。见https://github.com/tornadoweb/tornado/blob/master/demos/webspider/webspider.py

达到max_clients限制，请求排队龙卷风

1 个答案: