我正在学习python和scraping，并使用Scrapy编写了我的第一个蜘蛛。当我在本地运行它来刮我的测试site它工作正常时它工作正常。我在Scrapyd的远程服务器上部署了项目，但是当我安排蜘蛛对着测试站点运行时，它总是返回503。 起初我想也许我的服务器IP是防火墙的，但我已经改变了IP。 我试图欺骗我的用户代理来模仿我的一个浏览器，但没有效果。 我不确定我还能尝试什么，因为这对我来说是新的领域，但任何指针都会受到赞赏。 编辑： 蜘蛛日志的相关输出：
2017-07-28 18:49:45 [scrapy.core.engine] INFO: Spider opened
2017-07-28 18:49:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-07-28 18:49:45 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-07-28 18:49:45 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://pieces-auto.oscaro.com/batterie-2585-g> (failed 1 times): 503 Service Unavailable
2017-07-28 18:49:45 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://pieces-auto.oscaro.com/batterie-2585-g> (failed 2 times): 503 Service Unavailable
2017-07-28 18:49:45 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://pieces-auto.oscaro.com/batterie-2585-g> (failed 3 times): 503 Service Unavailable
2017-07-28 18:49:45 [scrapy.core.engine] DEBUG: Crawled (503) <GET https://pieces-auto.oscaro.com/batterie-2585-g> (referer: None)
2017-07-28 18:49:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <503 https://pieces-auto.oscaro.com/batterie-2585-g>: HTTP status code is not handled or not allowed
2017-07-28 18:49:45 [scrapy.core.engine] INFO: Closing spider (finished)
2017-07-28 18:49:45 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 829,
'downloader/request_count': 3,
'downloader/request_method_count/GET': 3,
'downloader/response_bytes': 62411,
'downloader/response_count': 3,
'downloader/response_status_count/503': 3,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 7, 28, 16, 49, 45, 953027),
'httperror/response_ignored_count': 1,
'httperror/response_ignored_status_count/503': 1,
'log_count/DEBUG': 5,
'log_count/INFO': 9,
'memusage/max': 81936384,
'memusage/startup': 81936384,
'response_received_count': 1,
'retry/count': 2,
'retry/max_reached': 1,
'retry/reason_count/503 Service Unavailable': 2,
'scheduler/dequeued': 3,
'scheduler/dequeued/memory': 3,
'scheduler/enqueued': 3,
'scheduler/enqueued/memory': 3,
'start_time': datetime.datetime(2017, 7, 28, 16, 49, 45, 639361)}