我在服务器上运行了许多刮板,其中几个刮板在几个小时后被我杀死,我不明白其原因。而其他刮板此时仍在运行。
我尝试了减少并发请求的选项,但是没有用
这是我的日志
2019-06-15 09:57:06,922 - logstats - scrapy.extensions.logstats - INFO - Crawled 10868 pages (at 9 pages/min), scraped 6318 items (at 9 items/min)
2019-06-15 09:58:06,922 - logstats - scrapy.extensions.logstats - INFO - Crawled 10879 pages (at 11 pages/min), scraped 6329 items (at 11 items/min)
2019-06-15 09:59:06,922 - logstats - scrapy.extensions.logstats - INFO - Crawled 10889 pages (at 10 pages/min), scraped 6339 items (at 10 items/min)
2019-06-15 10:00:06,922 - logstats - scrapy.extensions.logstats - INFO - Crawled 10899 pages (at 10 pages/min), scraped 6349 items (at 10 items/min)
Killed