Question

我正在关注this教程，试图学习如何使用scrapy。我目前正在做第二个教程，编写自定义蜘蛛。我创建了项目并编写了redditbot.py，正如教程指定的那样。

import scrapy

class RedditbotSpider(scrapy.Spider):
name = 'redditbot'
allowed_domains = ['reddit.com/r/gameofthrones/']
start_urls = ['http://www.reddit.com/r/gameofthrones//']

def parse(self, response):
    #Extracting the content using css selectors
    titles = response.css('.title.may_blank::text').extract()
    votes = response.css('.score.unvoted::text').extract()
    times = response.css('time::attr(title)').extract()
    comments = response.css('.comments::text').extract()

    #Display the extracted contect in row fashion
    for item in zip(titles,votes,times,comments):
        #Creates a dictionary to store the scraped info
        scraped_info = {
            'title' : item[0],
            'vote' : item[1],
            'created_at' : item[2],
            'comments' : item[3],
        }

        #Yield/give the scraped info to scrapy
        yield scraped_info

但是，当我使用

运行程序时

scrapy crawl redditbot

程序将运行但不输出任何已删除的数据，正如教程所说的那样。这是我在终端收到的输出：

evans-mbp:ourfirstscraper evanyamaguchi$ scrapy crawl redditbot
2018-01-04 13:24:53 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: ourfirstscraper)
2018-01-04 13:24:53 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.3.1, w3lib 1.18.0, Twisted 17.9.0, Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), cryptography 2.1.4, Platform Darwin-17.3.0-x86_64-i386-64bit
2018-01-04 13:24:53 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'ourfirstscraper', 'FEED_FORMAT': 'csv', 'FEED_URI': 'reddit.csv', 'NEWSPIDER_MODULE': 'ourfirstscraper.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['ourfirstscraper.spiders']}
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-01-04 13:24:53 [scrapy.core.engine] INFO: Spider opened
2018-01-04 13:24:53 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-01-04 13:24:53 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-01-04 13:24:53 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.reddit.com/robots.txt> from <GET http://www.reddit.com/robots.txt>
2018-01-04 13:24:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.reddit.com/robots.txt> (referer: None)
2018-01-04 13:24:53 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.reddit.com/r/gameofthrones//> from <GET http://www.reddit.com/r/gameofthrones//>
2018-01-04 13:24:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.reddit.com/r/gameofthrones//> (referer: None)
2018-01-04 13:24:53 [scrapy.core.engine] INFO: Closing spider (finished)
2018-01-04 13:24:53 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 945,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 4,
 'downloader/response_bytes': 36092,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/301': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 1, 4, 18, 24, 53, 755172),
 'log_count/DEBUG': 5,
 'log_count/INFO': 7,
 'memusage/max': 66359296,
 'memusage/startup': 66359296,
 'response_received_count': 2,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2018, 1, 4, 18, 24, 53, 205879)}
2018-01-04 13:24:53 [scrapy.core.engine] INFO: Spider closed (finished)

我无法弄清楚为什么蜘蛛似乎在运行，但没有从网站上删除任何数据。

提前致谢，埃文

Answer 1

看起来你只是有一个错字班级名称为may-blank，而不是may_blank。

scrapy脚本在没有（明显）错误的情况下运行，但不会刮取数据

1 个答案: