scrapy爬行0页（每分钟0页），刮掉0项（0件/分）

时间：2017-09-27 10:07:42

标签： python scrapy

我正在学习scrapy并希望从这个页面中搜索一些项目： https://www.gumtree.com/search?sort=date&search_category=flats-houses&q=box&search_location=Vale+of+Glamorgan

为了避免robots.txt政策等，我已经将页面保存在我的高清上，并使用scrapy shell测试了我的xpath。他们似乎按预期工作。但是当我使用scrapy crawl basic命令运行我的蜘蛛时（因为我在阅读的书中推荐它）我得到了以下输出：

    2017-09-27 12:05:02 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: properties)
2017-09-27 12:05:02 [scrapy.utils.log] INFO: Overridden settings: {'USER_AGENT': 'Mozila/5.0', 'SPIDER_MODULES': ['properties.spiders'], 'BOT_NAME': 'properties', 'NEWSPIDER_MODULE': 'properties.spiders'}
2017-09-27 12:05:03 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2017-09-27 12:05:03 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-09-27 12:05:03 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-09-27 12:05:03 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-09-27 12:05:03 [scrapy.core.engine] INFO: Spider opened
2017-09-27 12:05:03 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-09-27 12:05:03 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6026
2017-09-27 12:05:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET file:///home/albert/Documents/programming/python/scrapy/properties/properties/tests/test_page.html> (referer: None)
2017-09-27 12:05:04 [basic] DEBUG: title: 
2017-09-27 12:05:04 [basic] DEBUG: price: 
2017-09-27 12:05:04 [basic] DEBUG: description: 
2017-09-27 12:05:04 [basic] DEBUG: address: 
2017-09-27 12:05:04 [basic] DEBUG: image_urls: 
2017-09-27 12:05:04 [scrapy.core.engine] INFO: Closing spider (finished)
2017-09-27 12:05:04 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 262,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 270547,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 9, 27, 9, 5, 4, 91741),
 'log_count/DEBUG': 7,
 'log_count/INFO': 7,
 'memusage/max': 50790400,
 'memusage/startup': 50790400,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 9, 27, 9, 5, 3, 718976)}
2017-09-27 12:05:04 [scrapy.core.engine] INFO: Spider closed (finished)
igor@foobard:properties$ scrapy crawl basic
2017-09-27 12:10:13 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: properties)
2017-09-27 12:10:13 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_MODULES': ['properties.spiders'], 'BOT_NAME': 'properties', 'NEWSPIDER_MODULE': 'properties.spiders', 'USER_AGENT': 'Mozila/5.0'}
2017-09-27 12:10:13 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole']
2017-09-27 12:10:13 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-09-27 12:10:13 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-09-27 12:10:13 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-09-27 12:10:13 [scrapy.core.engine] INFO: Spider opened
2017-09-27 12:10:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-09-27 12:10:13 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6026
2017-09-27 12:10:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET file:///home/albert/Documents/programming/python/scrapy/properties/properties/tests/test_page.html> (referer: None)
2017-09-27 12:10:13 [basic] DEBUG: title: 
2017-09-27 12:10:13 [basic] DEBUG: price: 
2017-09-27 12:10:13 [basic] DEBUG: description: 
2017-09-27 12:10:13 [basic] DEBUG: address: 
2017-09-27 12:10:13 [basic] DEBUG: image_urls: 
2017-09-27 12:10:13 [scrapy.core.engine] INFO: Closing spider (finished)
2017-09-27 12:10:13 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 262,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 270547,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 9, 27, 9, 10, 13, 927817),
 'log_count/DEBUG': 7,
 'log_count/INFO': 7,
 'memusage/max': 51032064,
 'memusage/startup': 51032064,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 9, 27, 9, 10, 13, 722731)}
2017-09-27 12:10:13 [scrapy.core.engine] INFO: Spider closed (finished)

这是 items.py ：

from scrapy.item import Item, Field


class PropertiesItem(Item):
    title = Field()
    price = Field()
    description = Field()
    address = Field()
    image_urls = Field()

    images = Field()
    location = Field()

    url = Field()
    project = Field()
    spider = Field()
    server = Field()
    date = Field()

这是蜘蛛 basic.py ：

import scrapy


class BasicSpider(scrapy.Spider):
    name = 'basic'
    start_urls = ['file:///home/albert/Documents/programming/python/scrapy/properties/properties/site/test_page.html']

    def parse(self, response):
        self.log('title: '.format(response.xpath(
            "//h2[@class='listing-title' and not(span)]/text()").extract()))
        self.log('price: '.format(response.xpath(
            "//meta[@itemprop='price']/@content").extract()))
        self.log("description: ".format(response.xpath(
            "//p[@itemprop='description' and not(span)]/text()").extract()))
        self.log('address: '.format(response.xpath(
            "//span[@class='truncate-line']/text()[2]").re('\|(\s+\w+.+)')))
        self.log('image_urls: '.format(response.xpath(
            "//noscript/img/@src").extract()))

xpaths有点笨拙但它们有效。但是不收集这些物品。我想知道原因。

2 个答案:

答案 0 :(得分：1)

我不在本地文件中尝试Scrapy，但是如果你想要搜索某些东西，你必须首先初始化Items并且必须在{python}中将Item赋值为dict ，最后yield item到pipeline 

import scrapy
from properties.items import PropertiesItem

class BasicSpider(scrapy.Spider):
    name = 'basic'
    start_urls = ['file:///home/albert/Documents/programming/python/scrapy/properties/properties/site/test_page.html']

    def parse(self, response):
        item = PropertiesItem()     # init Item
        # assignment 
        item['title'] = response.xpath("//h2[@class='listing-title' and not(span)]/text()").extract()
        item['price'] = response.xpath("//h2[@class='listing-title' and not(span)]/text()").extract()
        item['description'] = response.xpath("//h2[@class='listing-title' and not(span)]/text()").extract()
        item['address'] = response.xpath("//h2[@class='listing-title' and not(span)]/text()").extract()
        item['image_urls'] = response.xpath("//h2[@class='listing-title' and not(span)]/text()").extract()
        # yield item
        yield item

答案 1 :(得分：1)

您的问题是您没有在字符串中的任何位置插入格式函数的输出。因此，您需要将title更改为title {}，因此格式会插入值。也可以使用extract_first()代替extract()。所以你得到一个字符串输出而不是数组

class BasicSpider(scrapy.Spider):
    name = 'basic'
    start_urls = ['file:///home/albert/Documents/programming/python/scrapy/properties/properties/site/test_page.html']

    def parse(self, response):
        self.log('title: {}'.format(response.xpath(
            "//h2[@class='listing-title' and not(span)]/text()").extract_first()))
        self.log('price: {}'.format(response.xpath(
            "//meta[@itemprop='price']/@content").extract_first()))
        self.log("description: {}".format(response.xpath(
            "//p[@itemprop='description' and not(span)]/text()").extract_first()))
        self.log('address: {}'.format(response.xpath(
            "//span[@class='truncate-line']/text()[2]").re('\|(\s+\w+.+)')))
        self.log('image_urls: {}'.format(response.xpath(
            "//noscript/img/@src").extract_first()))
相关问题
最新问题