Scrapy在抓取404 URL时在pymongo上报告错误

时间:2020-05-19 09:15:13

标签: scrapy

我已经部署了scrapy项目来抓取一些页面,而一些404页会引发一些错误,这些错误会重新添加到pymongo中,这是日志:

2020-05-19 16:54:13 [scrapy.core.scraper] ERROR: Spider error processing <GET https://repo.packagist.org/p/dehdjdnje.json> (referer: None)
Traceback (most recent call last):
  File "/home/chengwei/PycharmProjects/scrapytest/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/chengwei/PycharmProjects/scrapytest/venv/lib/python3.6/site-packages/scrapy/core/spidermw.py", line 81, in process_spider_exception
    result = method(response=response, exception=exception, spider=spider)
  File "/home/chengwei/PycharmProjects/scrapytest/venv/lib/python3.6/site-packages/scrapy/spidermiddlewares/httperror.py", line 49, in process_spider_exception
    spider.crawler.stats.inc_value('httperror/response_ignored_count')
  File "/home/chengwei/PycharmProjects/scrapytest/venv/lib/python3.6/site-packages/pymongo/collection.py", line 3343, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'inc_value' method on a 'Collection' object it is failing because no such method exists.

我想问一下为什么错误与pymongo有关,以及如何处理。我的拼凑版本是2.1.0。

感谢您的回音。

0 个答案:

没有答案
相关问题