使用scrapy抓取网页时,请将代理设置如下:
LEFT JOIN
我抓住了 错误:
def process_request(self, request, spider):
proxy_address = random.choice(self.proxies.keys())
proxy_user_pass = self.proxies[proxy_address]
request.meta['proxy'] = proxy_address
basic_auth = 'Basic ' + base64.encodestring(proxy_user_pass)
request.headers['Proxy-Authorization'] = basic_auth
没有设置代理,它可以正常运行。