我写了一个简单的测试来验证scrapy中的https代理,但是没有用
class BaiduSpider(scrapy.Spider):
name = 'baidu'
allowed_domains = ['baidu.com']
start_urls = ['http://www.baidu.com/']
def parse(self, response):
if response.status == 200:
print(response.text)
以及类似以下的中间件文件: 类DynamicProxyDownloaderMiddleware(object):
def process_request(self, request, spider):
request.meta['proxy'] = 'https://183.159.88.182:8010'
也是设置文件:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'requestTest.middlewares.DynamicProxyDownloaderMiddleware': 100
}
使用请求库时,https代理可以工作,但是变成了scrapy。它使我感到困惑。所以有人知道吗?
日志文件:
[the log file][1]
[scrapy.downloadermiddlewares.retry]调试:重试http://www.baidu.com/>(失败1次):TCP连接超时:10060: 代理地址为https://183.159.88.182:8010