这是在网络抓取中旋转代理的正确方法吗

时间:2019-04-01 00:04:31

标签: python python-3.x web-scraping

我正在尝试在网络抓取中从一组proxy_pool中旋转代理。通过设置page_response = ""这样做是正确的方法吗?

page_reponse = ''
while page_reponse == '':
    proxy = next(proxy_pool)
    dict_proxy = {"http": proxy, "https": proxy}
    try:
        page_response = requests.get(iurl, headers=headers,timeout=tout, proxies=dict_proxy)
        break
    except requests.exceptions.ConnectionError as rec:
        pass

如何在不使用硒的情况下在代码中实现以下语句:

try a proxy for 3 times and if still it gives error then only i should go to next proxy in the list

0 个答案:

没有答案