Question

我知道这个问题是Scrapy ERROR: Error downloading - Could not open CONNECT tunnel

但是我找不到解决方案，因为我没有足够的声誉，我无法在那里发表评论并寻求答案。

所以这是我的settings.py文件代码

 DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'clientemails.middlewares.ProxyMiddleware': 100,
}   


PROXIES = [{'ip_port': '74.118.91.233:8080', 'user_pass': ''},
           {'ip_port': '83.222.222.137:8080', 'user_pass': ''},
           {'ip_port': '197.89.196.48:8080', 'user_pass': ''},
           {'ip_port': '112.96.29.187:80', 'user_pass': ''},
           {'ip_port': '202.194.101.150:80', 'user_pass': ''},
           {'ip_port': '213.85.92.10:80', 'user_pass': ''},
           {'ip_port': '221.208.194.108:80', 'user_pass': ''},
           {'ip_port': '190.63.140.71:80', 'user_pass': ''},
           {'ip_port': '223.19.196.232:80', 'user_pass': ''},
           {'ip_port': '195.71.127.224:80', 'user_pass': ''},]

而且middlewares.py有这个代码

import base64
import random
from settings import PROXIES

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        proxy = random.choice(PROXIES)
        if proxy['user_pass'] is not None:
            request.meta['proxy'] = "http://%s" % proxy['ip_port']
            encoded_user_pass = base64.encodestring(proxy['user_pass'])
            request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass            
        else:
            request.meta['proxy'] = "http://%s" % proxy['ip_port']

当我运行机器人时，我收到此错误：

 Could not open CONNECT tunnel.

任何帮助都将不胜感激。

Answer 1

不要在标题中传递您的代理授权，而是尝试格式化您的请求：

request.meta['proxy'] = "http://{}:{}@{}:{}".format(user,pass,'127.0.0.1','8118')

Scrapy ERROR：下载<get>时出错无法打开CONNECT隧道

1 个答案: