Python异步Web请求

时间:2019-07-02 15:45:06

标签: python-asyncio python-3.7 aiohttp

我正在尝试从充满公司名称的数据文件中向网页提出异步请求

The code to make requests
async def fetch(company):

    #Create Query
    #url = format_text(company)
    params = str(company).replace(" ", "+")

    async with aiohttp.ClientSession() as session:
        async with session.get('https://www.hitta.se/sök?vad=', params=params) as page:
            print(f"Getting {company}")
            #await page.text()
            if page.status_code == 200:
                strip_website(page, company)

The code to run thought all company names:
async def scrape():

    #Companies to scrape
    companies = getData()
    tasks = []

    #loop = asyncio.get_event_loop()
    for company in companies:
        try:
            #task = await fetch(company)
            print(f"Adding {company} to task")
            task = asyncio.create_task(fetch(company))
            tasks.append(task)
        except Exception as e:
            print(e)

    responses  =  asyncio.gather(*tasks)


The code to start the downloads

def start_download():
    print("Start downloading data")
    loop = asyncio.get_event_loop()

    results = loop.run_until_complete(scrape())
    #asyncio.run(scrape())
start_download()

我希望返回该网页,该网页将传递到strip_site()以仅获取我需要的数据。     此刻,里面的代码块永远不会运行。

async with session.get('https://www.hitta.se/sök?vad=', params=params) as page:

0 个答案:

没有答案