Question

我正在关注这个scrapy介绍： https://doc.scrapy.org/en/1.4/intro/overview.html

代码运行良好，直到需要跳转到下一页，然后我得到以下错误堆栈：

2017-08-09 19:45:37 [scrapy.core.scraper] ERROR: Spider error processing <GET http://quotes.toscrape.com/tag/humor/> (referer: None)

Traceback (most recent call last):
  File "/Users/totosh/anaconda/lib/python3.5/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/Users/totosh/anaconda/lib/python3.5/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/Users/totosh/anaconda/lib/python3.5/site-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/Users/totosh/anaconda/lib/python3.5/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Users/totosh/anaconda/lib/python3.5/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/Users/totosh/OneDrive/data analysis and ML/python working directory/scrapy0.py", line 48, in parse
    yield response.follow(next_page, self.parse)
AttributeError: 'HtmlResponse' object has no attribute 'follow'

是什么原因造成的？

是因为html是：<a href="/tag/humor/page/2/">Next <span aria-hidden="true">→</span></a>并且没有绝对网址吗？

谢谢

Answer 1

您使用的是什么版本的Scrapy？ Scrapy 1.4.0中添加了follow个Response对象方法 - 请参阅Release notes。

Scrapy response.follow抛出错误

1 个答案: