Question

我正在尝试遵循this指南：对API进行大量调用。

我有一个异步任务管理器，如下所示：

class TaskPool(object):

    def __init__(self, workers):
        self._semaphore = asyncio.Semaphore(workers)
        self._tasks = set()


    async def put(self, coro):

        await self._semaphore.acquire()

        task = asyncio.ensure_future(coro)
        self._tasks.add(task)
        task.add_done_callback(self._on_task_done)

    def _on_task_done(self, task):
        self._tasks.remove(task)
        self._semaphore.release()

    async def join(self):
        x = asyncio.gather(*self._tasks)
        return x

    async def __aenter__(self):
        return self

    def __aexit__(self, exc_type, exc, tb):
        print("This_is_returning something")
        return self.join()

然后我有一个如下所示的访存和主要方法：

async def fetch(url, session):
    async with session.get(url) as response:
        return await response.read()

async def _main(url, total_requests):
    connector = TCPConnector(limit=None)
    async with ClientSession(connector=connector) as session, TaskPool(limit) as tasks:

        for i in range(total_requests):
            await tasks.put(fetch(url.format(i), session)

        t = tasks.join()

    return await t

并且正在像这样使用它：

def make_batch_of_calls(calls, limit):
    loop = asyncio.get_event_loop()
    test = loop.run_until_complete(_main(calls, limit= limit))
    return test

list_of_urls = ( "http://localhost:8080/{}".format(i) for i in range(int(10000000)) )

data = list_of_urls[:100]
sublists = [data[x:x+10] for x in range(0, len(data), 10)]

for i in range(len(sublists)):
    r = make_batch_of_calls(sublists[i], limit=5)
    with open(tmp_folder_path + "/responses_{n}.json".format(n=i), 'w') as output_file:
        json.dump(r, output_file)
    print("{n}/{t}".format(n = i, t = len(sublists)))

我的问题是我无法获取它返回的所有响应。好像正在进行所有调用，因为我可以在API端看到所有传入的请求，但是我在JSON文件中读取的响应只是限制。该限制应仅限制建立的连接数，而不限制其持有的响应数。

如果我严格按照上述指南进行操作，我将一无所获。

有没有办法让我拨打所有电话并返回所有回复？

谢谢

如何获得一个异步任务管理器以返回协程执行的所有结果

0 个答案: