Question

我有2个网址和6万多个请求。基本上，我需要将每个请求都发布到两个URL，然后比较它们的响应，但不必等待响应发布另一个请求。

我尝试使用aiohttp和asyncio

import asyncio
import time
import aiohttp
import os

from aiofile import AIOFile

testURL = ""
prodURL = ""

directoryWithRequests = ''
directoryToWrite = ''

headers = {'content-type': 'application/soap+xml'}

i = 1


async def fetch(session, url, reqeust):
    global i
    async with session.post(url=url, data=reqeust.encode('utf-8'), headers=headers) as response:
        if response.status != 200:
            async with AIOFile(directoryToWrite + str(i) + '.xml', 'w') as afp:
                await afp.write(reqeust)
                i += 1
        return await response.text()


async def fetch_all(session, urls, request):
    results = await asyncio.gather(*[asyncio.create_task(fetch(session, url, request)) for url in urls])
    return results


async def asynchronousRequests(requestBody):
    urls = [testURL, prodURL]
    global i
    with open(requestBody) as my_file:
        body = my_file.read()

    async with aiohttp.ClientSession() as session:
        htmls = await fetch_all(session, urls, body)
        # some conditions

async def asynchronous():
    try:
        start = time.time()
        futures = [asynchronousRequests(directoryWithRequests + i) for i in os.listdir(directoryWithRequests)]
        for future in asyncio.as_completed(futures):
            result = await future

        print("Process took: {:.2f} seconds".format(time.time() - start))

    except Exception as e:
        print(str(e))

if __name__ == '__main__':
    try:
        # AsyncronTest
        ioloop = asyncio.ProactorEventLoop()
        ioloop.run_until_complete(asynchronous())
        ioloop.close()
        if i == 1:
            print('Regress is OK')
        else:
            print('Number of requests to check = {}'.format(i))

    except Exception as e:
        print(e)

我相信上面的代码有效，但是它创建了N个Future，其中N等于请求文件的数量。 ddos之所以出现，是因为服务器无法同时响应该数量的请求。

Answer 1

找到合适的解决方案。基本上，这只是两个异步任务：

tasks = [
                postRequest(testURL, client, body),
                postRequest(prodURL, client, body)
            ]
await asyncio.wait(tasks)

与请求中的请求数量不一样的问题的代码性能不同，但是至少它不会使服务器产生太多的麻烦。

如何使用aiohttp将请求组发布到2个URL

1 个答案: