我在localhost上托管服务器,我希望异步发出数百个GET请求。为此我使用grequests
。一切出现以便正常工作但我反复收到警告:
警告:requests.packages.urllib3.connectionpool:连接池已满,丢弃连接:date.jsontest.com
搜索显示在Session()
中创建requests
时如何避免完整池问题,例如here。但是,有几件事:
pool_maxsize
数量的任何请求都会发出警告。requests.packages.urllib3.disable_warnings()
似乎无能为力。所以我的问题是:
grequests
库实际相关,尤其是当我采取措施限制池大小时?我是否会在测试中引发意外行为并使我的预期结果失效?要测试的一些代码:
import grequests
import requests
requests.packages.urllib3.disable_warnings() # Doesn't seem to work?
session = requests.Session()
# Hashing the below will cause 105 warnings instead of 5
adapter = requests.adapters.HTTPAdapter(pool_connections=100,
pool_maxsize=100)
session.mount('http://', adapter)
# Test query
query_list = ['http://date.jsontest.com/' for x in xrange(105)]
rs = [grequests.get(item, session=session) for item in query_list]
responses = grequests.map(rs)
print len([item.json() for item in responses])