Question

我正在尝试一个小功能，从JSON端点抓取数据，

该网址就像https://xxxxxxxx.com/products.json?&page=“，我可以插入一个页码，

当我使用请求模块时，我有一个while循环，增加页码并中断直到得到一个空的响应（该页面为空）

是否可以使用aiohttp做同样的事情？

到目前为止，我仅实现了预生成一定数量的网址并将其传递给任务想知道我是否也可以使用循环并在看到空响应时停止

非常感谢您

'''

import asyncio
import aiohttp


async def download_one(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            pprint.pprint(await resp.json(content_type=None))


async def download_all(sites):
    tasks = [asyncio.create_task(download_one(site)) for site in sites]
    await asyncio.gather(*tasks)

def main():
    sites = list(map(lambda x: request_url + str(x), range(1, 50)))
    asyncio.run(download_all(sites))

'''

Answer 1

这是一段 unested 代码。即使不起作用，它也会为您提供一个完成工作的方法

import asyncio
import aiohttp


async def download_one(session, url):
    async with session.get(url) as resp:
        resp = await resp.json()
        if not resp:
            raise Exception("No data found") # needs to be there for breaking the loop


async def download_all(sites):
    async with aiohttp.ClientSession() as session:
        futures = [download_one(session, site) for site in sites]
        done, pending = await asyncio.wait(
            futures, return_when=FIRST_EXCEPTION  # will return the result when exception is raised by any future
        )

        for future in pending:
            future.cancel()  # it will shut down all redundant jobs

def main():
    sites = list(map(lambda x: request_url + str(x), range(1, 50)))
    asyncio.run_until_complete(download_all(sites))

Aiohttp尝试获取具有页码的页面响应，直到击空响应

1 个答案: