Question

我的Scrapy代码不起作用，我不确定为什么。我的蜘蛛是对Reddit上的权力游戏subreddit进行爬网的测试。

这是我的代码：

import scrapy


class Redditbot2Spider(scrapy.Spider):
    name = 'redditbot2'
    allowed_domains = ['www.reddit.com']
    start_urls = ['https://www.reddit.com/r/gameofthrones/']


def parse(self, response):
    titles = response.selector.xpath('//h2/text()').extract()
    votes = response.selector.xpath('//div[@class="_1rZYMD_4xY3gRcSS3p80D0"]/test()').extract()
    time = response.selector.xpath('//a[@class="_3jOxDPIQ0KaOWpzvSQo-1s"]/text()').extract()
    comments = response.selector.xpath('//span[@class="FHCV02u6Cp2zYL0fhQPsO"])/text()').extract()

    for item in zip(titles, votes, time, comments):
        scraped_info = {
            'title': titles[0],
            'vote': votes[1],
            'time': time[2],
            'comments': comments[3],
        }
        yield scraped_info

这是错误的日志：

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py", line 90, in parse
    raise NotImplementedError('{}.parse callback is not defined'.format(self.__class__.__name__))
NotImplementedError: Redditbot2Spider.parse callback is not defined

Redditbot2Spider.parse方法在那里，所以我不明白为什么这么说。有什么想法吗？

Answer 1

您的代码存在缩进问题，parse方法与类处于同一级别，因此解释器不会意识到它是该类的成员。您必须在parse方法中进行缩进：

class Redditbot2Spider(scrapy.Spider): 
    name = 'redditbot2'
    allowed_domains = ['www.reddit.com']
    start_urls = ['https://www.reddit.com/r/gameofthrones/']
    def parse(self, response): 
        titles = response.selector.xpath('//h2/text()').extract()
        #etc.

Scrapy显示notImplementedError，我不知道为什么

1 个答案: