Question

我正在尝试使用Python，并试图构建简单的Google Scraper。为此，我尝试实现代理，但是我没有足够的知识。

到目前为止，我的代码如下：

import requests
from bs4 import BeautifulSoup as bs
import warnings
from googlesearch import search
from itertools import cycle

warnings.filterwarnings("ignore", module='bs4')

proxies = {
'http:myproxies_url'

}
proxy_pool = cycle(proxies)
proxy = next(proxy_pool)

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0',
'Accept': '*/*',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'X-Requested-With': 'XMLHttpRequest',
'Connection': 'keep-alive'}

def searchBing(query, num):

    urls = []

    for j in search(query, tld="co.in", num=10, stop=10, pause=3): 
            proxies = {
                "http": proxy,
                "https":proxy
            }
            urls.append(j) 


    return urls[:num]

def extractText(url):
    page = requests.get(url,proxies=proxies)
    soup = bs(page.text, 'html.parser')
    return soup.get_text()

这是控制台错误：

no_proxy = proxies.get('no_proxy') if proxies is not None else None
AttributeError: 'set' object has no attribute 'get'

Answer 1

proxies是set，

proxies = {
   'http:myproxies_url'
}

该错误向我提示，没有通过searchBing函数将其设置为字典。我建议更新您的变量名，以免发生这种情况。

如何使用Python旋转代理

1 个答案: