连接因scrapy,privoxy和tor而被拒绝

时间:2017-02-14 07:04:55

标签: curl proxy scrapy tor

我的ubuntu 16.04盒子上安装并配置了Privoxy,tor和scrapy 1.3.1。当我在终端中输入以下内容时,我可以看到privoxy / tor正在运行

export http_proxy="http://localhost:8118"
curl ipecho.net/plain; echo

我从ipecho.net获得的IP与我的不同。

当我运行一个与上面的卷曲相同的测试蜘蛛并且没有启用代理时,一切运行良好,我从ipecho.net得到答案。

当我添加代理

export http_proxy="http://localhost:8118"
scrapy crawl test

导致错误消息:

2017-02-14 07:45:36 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://ipecho.net/robots.txt> (failed 1 times): Connection was refused by other side: 111: Connection refused.
2017-02-14 07:45:36 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://ipecho.net/robots.txt> (failed 2 times): Connection was refused by other side: 111: Connection refused.
2017-02-14 07:45:36 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://ipecho.net/robots.txt> (failed 3 times): Connection was refused by other side: 111: Connection refused.
2017-02-14 07:45:36 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://ipecho.net/robots.txt>: Connection was refused by other side: 111: Connection refused.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1297, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python3.5/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python3.5/dist-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused

知道scrapy / twisted出了什么问题吗?

我也尝试过request.meta [&#39; proxy&#39;] =&#34; http://localhost:8118&#34;要么 代理类。

0 个答案:

没有答案