我正在尝试使用aiohttp进行http调用。我可以获得的最大吞吐量是1K请求/秒。服务器没有进行任何处理，只是立即处理请求。请求的RTT大约需要7毫秒。
python进程的分析显示了这样的东西。我试图异步地发出2K请求，他们花了2秒多才完成。
ncalls tottime percall cumtime percall filename:lineno(function)
3720 0.905 0.000 0.905 0.000 {method 'poll' of 'select.epoll' objects}
2000 0.142 0.000 0.158 0.000 {method 'feed_data' of 'aiohttp._http_parser.HttpParser' objects}
11500 0.081 0.000 1.958 0.000 grpc.py:113(request)
9500 0.080 0.000 1.760 0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:180(_request)
3720 0.069 0.000 3.634 0.001 /usr/lib/python3.6/asyncio/base_events.py:1330(_run_once)
4000 0.064 0.000 0.064 0.000 {method 'send' of '_socket.socket' objects}
4000 0.063 0.000 0.067 0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:635(_prepare_headers)
6000 0.053 0.000 0.215 0.000 /home/me/venv2/lib/python3.6/site-packages/idna/core.py:231(check_label)
54000 0.053 0.000 0.104 0.000 /home/me/venv2/lib/python3.6/site-packages/idna/intranges.py:38(intranges_contain)
26509 0.050 0.000 2.608 0.000 /usr/lib/python3.6/asyncio/events.py:125(_run)
5500 0.045 0.000 0.364 0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/connector.py:723(_create_connection)
182134/182126 0.045 0.000 0.067 0.000 {built-in method builtins.isinstance}
1500 0.042 0.000 0.042 0.000 {method 'connect' of '_socket.socket' objects}
2000 0.040 0.000 0.178 0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client_reqrep.py:477(send)
382 0.040 0.000 0.040 0.000 {built-in method marshal.loads}
9500 0.034 0.000 2.083 0.000 grpc.py:65(grpc)
如何提高吞吐量？
这是我的实施
def _get_session_headers() -> dict:
"""Get session headers."""
return {
// Some key value for headers
}
conn = aiohttp.TCPConnector(limit=1000)
_session = aiohttp.ClientSession(connector=conn,headers=_get_session_headers())
def _get_uri(path: str) -> str:
"""Get URI."""
return urljoin("http://targethost:8080/", path)
async def make_request(
path: str,
grequest: Message,
index, int
headers: dict = {},
**kwargs: dict) -> None:
async with request('POST', _get_uri(path), data=grequest, headers=headers, **kwargs) as response:
print("Before " + str(index))
buffer = await response.read()
print("After " + str(index))
@async_contextmanager
async def request(method: str, path: str, **kwargs: dict) -> aiohttp.ClientResponse:
async with _session.request(method, _get_uri(path), **kwargs) as res:
yield res
async def test(req, index):
await make_request('fetch', req, index)
grequest = initialize_data() // Some sample data to be sent
start = time.time()
loop = asyncio.get_event_loop()
tasks = []
start = time.time()
for i in range(1000):
tasks.append(asyncio.ensure_future(test(grequest, i)))
print("Starting the loop")
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
task=loop.create_task(_session.close())
loop.run_until_complete(task)
loop.close()
print(end-start)
打印时间为1秒。所以我的rps只有1k / sec。
Before buffer 896
After buffer 896
Before buffer 976
After buffer 976
Before buffer 880
After buffer 880
Before buffer 898
After buffer 898
Before buffer 882
After buffer 882
Before buffer 978
After buffer 978
Before buffer 900
After buffer 900
Before buffer 884
After buffer 884
Before buffer 982
After buffer 982
Before buffer 902
After buffer 902
Before buffer 886
After buffer 886
Before buffer 904
After buffer 904
Before buffer 988
After buffer 988
Before buffer 888
After buffer 888
Before buffer 990
After buffer 990
Before buffer 994
After buffer 994
看起来他们正在连续执行。但是，如果我更换
buffer = await response.read() buffer = await asyncio.sleep(1) import asyncio
from aiohttp import ClientSession
async def fetch(url, session, index):
async with session.get(url) as response:
print("Before " + str(index))
buffer = await response.read()
print("After " + str(index))
async def run(r):
url = "http://google.com"
tasks = []
# Fetch all responses within one Client session,
# keep connection alive for all requests.
async with ClientSession() as session:
for i in range(r):
task = asyncio.ensure_future(fetch(url, session, i))
tasks.append(task)
responses = await asyncio.gather(*tasks)
# you now have all response bodies in this variable
print(responses)
def print_responses(result):
print(result)
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(4))
loop.run_until_complete(future)
，版画混合在一起。
我还验证了本教程脚本的情况
ncalls tottime percall cumtime percall filename:lineno(function)
35901 10.792 0.000 10.792 0.000 {method 'poll' of 'select.epoll' objects}
20000 0.607 0.000 0.607 0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:635(_prepare_headers)
35901 0.397 0.000 22.202 0.001 /usr/lib/python3.6/asyncio/base_events.py:1330(_run_once)
33500 0.393 0.000 7.911 0.000 /home/me/venv2/lib/python3.6/site-packages/aiohttp/client.py:180(_request)
63007 0.345 0.000 0.430 0.000 /usr/lib/python3.6/asyncio/base_events.py:594(_call_soon)
20000 0.341 0.000 0.341 0.000 {method 'send' of '_socket.socket' objects}
10000 0.311 0.000 0.311 0.000 {method 'copy' of 'multidict._multidict.CIMultiDict' objects}
270000 0.282 0.000 0.557 0.000 /home/me/venv2/lib/python3.6/site-packages/idna/intranges.py:38(intranges_contain)
30000 0.263 0.000 1.103 0.000 /home/me/venv2/lib/python3.6/site-packages/idna/core.py:231(check_label)
10000 0.246 0.000 0.355 0.000 {method 'feed_data' of 'aiohttp._http_parser.HttpParser' objects}
对于10000个请求，这是我从探查器
获得的{{1}}