我将Scrapy与代理/渲染服务“ scraperapi”结合使用。
当我像这样(https://api.scraperapi.com?key=d2debdcfbd1854e4505cd5c7ac74a177&url=https://www.example.com)在Chrome或Postman中对服务运行GET请求时,它运行正常,但所有scrapy请求都失败,并出现以下错误。
这是Scrapy / Twisted错误还是我做错了什么?
[scrapy.core.scraper] ERROR: Error downloading <GET
https://api.scraperapi.com?key=d2debdcfbd1854e4505cd5c7ac74a177&url=https://www.example.com>
Traceback (most recent call last):
File "scrapy\core\downloader\middleware.py", line 43, in process_request defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]