Question

我需要在我的Scrapy项目中配置代理。当我尝试运行它时，它将引发以下错误：

错误

2019-05-09 19:36:50 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-05-09 19:36:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.myip.com/> (failed 1 times): Could not open CONNECT tunnel with proxy x.x.x.x:xxxx [{'status': 407, 'reason': b'Unauthorized'}]

代码

class FirstSpyder(CrawlSpider):
    # The name of the spider
    name = "FirstSpyder"
    def __init__(self, *args, **kwargs):
        super(FirstSpyder, self).__init__(*args, **kwargs)
        self.proxy_pool = ['http://x.x.x.x:xxxx']

    # The domains that are allowed (links to other domains are skipped)
    allowed_domains = ["myip.com"]

    # The URLs to start with
    start_urls = ['https://www.myip.com/']

    def start_requests(self):
        for url in self.start_urls:
            request = Request(url, dont_filter=True, callback=self.parse)
            # set the meta['item'] to use the item in the next call back
            request.meta['proxy'] = random.choice(self.proxy_pool)
            yield request

    def parse(self, response):
        ip = response.css('#ip::text').get()
        print ("IPV4::",ip)

Answer 1

从错误返回中可以看出，它显示为“未授权”。

状态代码407代表：“ HTTP 407必需的Proxy Authentication客户端错误状态响应代码指示该请求尚未应用，因为它缺少浏览器和服务器之间的代理服务器的有效身份验证凭据。可以访问请求的资源。“

没有看到一点代码，就帮不上忙...

有一个库可以帮助您做到这一点： https://github.com/aivarsk/scrapy-proxies

Scrapy中的代理配置

1 个答案: