Question

我正在尝试创建简单的Web监控脚本，该脚本定期和异步地将GET请求发送到列表中的URL。这是我的请求功能：

def request(url,timeout=10):
    try:
        response = requests.get(url,timeout=timeout)
        response_time = response.elapsed.total_seconds()
        if response.status_code in (404,500):
            response.raise_for_status()
        html_response = response.text
        soup = BeautifulSoup(html_response,'lxml')
        # process page here
        logger.info("OK {}. Response time: {} seconds".format(url,response_time))
    except requests.exceptions.ConnectionError:
        logger.error('Connection error. {} is down. Response time: {} seconds'.format(url,response_time))
    except requests.exceptions.Timeout:
        logger.error('Timeout. {} not responding. Response time: {} seconds'.format(url,response_time))
    except requests.exceptions.HTTPError:
        logger.error('HTTP Error. {} returned status code {}. Response time: {} seconds'.format(url,response.status_code, response_time))
    except requests.exceptions.TooManyRedirects:
        logger.error('Too many redirects for {}. Response time: {} seconds'.format(url,response_time))
    except:
        logger.error('Content requirement not found for {}. Response time: {} seconds'.format(url,response_time))

在这里我为所有网址调用此函数：

def async_requests(delay,urls):
    for url in urls:
        async_task = make_async(request,delay,url,10)
        loop.call_soon(delay,async_task)
    try:
        loop.run_forever()
    finally:
        loop.close()

delay参数是循环的间隔，它描述了需要执行函数的频率。为了循环request，我创建了这样的东西：

def make_async(func,delay,*args,**kwargs):

    def wrapper(*args, **kwargs):
        func(*args, **kwargs)
        loop.call_soon(delay, wrapper)

    return wrapper

每次执行async_requests时，我都会为每个网址收到此错误：

Exception in callback 1.0(<function mak...x7f1d48dd1730>)
handle: <Handle 1.0(<function mak...x7f1d48dd1730>)>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run
    self._callback(*self._args)
TypeError: 'float' object is not callable

每个网址的request函数也没有按预期定期执行。此外，async_requests之后的打印功能也未执行：

async_requests(args.delay,urls)
print("Starting...")

我知道我在代码中做错了但我无法解决如何解决这个问题。我是python的初学者，对asyncio不是很有经验。总结我想要达成的目标：

在不阻塞主线程的情况下，针对特定网址异步运行并定期request。
异步运行async_requests，这样我就可以启动一个简单的http服务器例如在同一个帖子中。

Answer 1

except:

它还会捕获服务例外行KeyboardInterrupt或StopIteration。永远不要这样做。而是写：

except Exception:

如何使用asyncio在Python 3中异步运行requests.get？

requests.get本质上是封锁的。

您应该找到aiohttp module：

等请求的异步替代方案

async def get(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            return await resp.text()

或在单独的线程中运行requests.get并使用loop.run_in_executor等待此线程异步性：

executor = ThreadPoolExecutor(2)

async def get(url):
    response = await loop.run_in_executor(executor, requests.get, url)
    return response.text

如何使用asyncio在Python 3中异步运行requests.get？

1 个答案: