使用Scrapy从刮痧表中刮取数据但不刮痧物品

时间：2016-05-18 02:38:04

标签： python scrapy

我第一次尝试Scrapy。经过相当多的研究，我得到了基础知识。现在我试图获取表的数据。它不起作用。 没有删除任何数据。请查看下面的源代码。

settings.py 

BOT_NAME = 'car' 
SPIDER_MODULES = ['car.spiders']
NEWSPIDER_MODULE ='car.spiders' 
DEFAULT_ITEM_CLASS = 'car.items.Car58Item'
ITEM_PIPELINES = {'car.pipelines.JsonLinesItemExporter': 300}

items.py 

from scrapy.item  import Item,Field
class Car58Item(Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    url = Field() 
    tip = Field() 
    name = Field() 
    size = Field() 
    region = Field() 
    amt = Field()

car_spider.py 

# -*- coding=utf-8 -*-
from __future__ import absolute_import
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider,Rule,Spider
from car.items import Car58Item


class CarSpider (CrawlSpider):
    name ='car'
    allowed_domains = ['58.com']
    start_urls = ['http://quanguo.58.com/ershouche']
    rules = [Rule(LinkExtractor(allow=('/pn\d+')),'parse_item')] #//页面读取策略

def parse_item(self,response):
    trs = response.xpath("//div[@id='infolist']/table[@class='tbimg']/tr")[1:-2]
    items = []
    #for tr in sel.xpath("id('infolist')/table/tr"):
    for tr in trs:
        item = Car58Item()
        item['url'] = tr.xpath("td[@class='img']/a/@href").extract()
        item['tip'] = tr.xpath("td[@class='t']/a/font/text()").extract()
        item['name'] = tr.xpath("td[@class='t']/a[1]/text()").extract()
        item['size'] = tr.xpath("td[@class='t']/p").extract()
        item['region'] = tr.xpath("td[@class='tc']/a/text()").extract()
        item['amt'] = tr.xpath("td[@class='tc']/b/text()").extract()
        items.append(item)
    return items

pipelines.py 

# -*- coding: utf-8 -*-
import json
import codecs

class JsonLinesItemExporter(object):
    def __init__(self):
       self.file = codecs.open('car.json','w',encoding='utf-8')

    def process_item(self, items, spider):
        line = json.dumps(dict(items),ensure_ascii=False) + "\n"
        self.file.write(line)
        return items

    def spider_closed(self,spider):
        self.file.close()

我在shell中运行scrapy 

 [mayuping@i~/PycharmProjects/car] $**scrapy crawl car**

2016-05-18 10:35:36 [scrapy] INFO: Scrapy 1.0.6 started (bot: car)
2016-05-18 10:35:36 [scrapy] INFO: Optional features available: ssl, http11
2016-05-18 10:35:36 [scrapy] INFO: Overridden settings: {'DEFAULT_ITEM_CLASS': 'car.items.Car58Item', 'NEWSPIDER_MODULE': 'car.spiders', 'SPIDER_MODULES': ['car.spiders'], 'BOT_NAME': 'car'}
2016-05-18 10:35:36 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2016-05-18 10:35:36 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-05-18 10:35:36 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-05-18 10:35:36 [scrapy] INFO: Enabled item pipelines: JsonLinesItemExporter
2016-05-18 10:35:36 [scrapy] INFO: Spider opened
2016-05-18 10:35:36 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-05-18 10:35:36 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-05-18 10:35:37 [scrapy] DEBUG: Redirecting (301) to <GET http://quanguo.58.com/ershouche/> from <GET http://quanguo.58.com/ershouche>
2016-05-18 10:35:39 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/> (referer: None)
2016-05-18 10:35:40 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn2/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn7/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn6/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn12/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:43 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn11/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:44 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn9/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:45 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn8/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:45 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn5/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn10/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn4/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn3/> (referer: http://quanguo.58.com/ershouche/)
2016-05-18 10:35:47 [scrapy] INFO: Closing spider (finished)
2016-05-18 10:35:47 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 5550,
 'downloader/request_count': 13,
 'downloader/request_method_count/GET': 13,
 'downloader/response_bytes': 339809,
 'downloader/response_count': 13,
 'downloader/response_status_count/200': 12,
 'downloader/response_status_count/301': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 5, 18, 2, 35, 47, 45187),
 'log_count/DEBUG': 14,
 'log_count/INFO': 7,
 'request_depth_max': 1,
 'response_received_count': 12,
 'scheduler/dequeued': 13,
 'scheduler/dequeued/memory': 13,
 'scheduler/enqueued': 13,
 'scheduler/enqueued/memory': 13,
 'start_time': datetime.datetime(2016, 5, 18, 2, 35, 36, 733155)}
2016-05-18 10:35:47 [scrapy] INFO: Spider closed (finished)

但不扫描任何数据... 

[mayuping@i~/PycharmProjects/car] $more car.json

car.json中的零项输出。

感谢

1 个答案:

答案 0 :(得分：1)

我的问题解决了。 parse_item未在car_spider.py中缩进。缩进后：

# -*- coding=utf-8 -*-
from __future__ import absolute_import
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider,Rule,Spider
from car.items import Car58Item


class CarSpider (CrawlSpider):
    name ='car'
    allowed_domains = ['58.com']
    start_urls = ['http://quanguo.58.com/ershouche']
    rules = [Rule(LinkExtractor(allow=('/pn\d+')),'parse_item')] #//页面读取策略

    def parse_item(self,response):
        trs = response.xpath("//div[@id='infolist']/table[@class='tbimg']/tr")[1:-2]
        items = []
        #for tr in sel.xpath("id('infolist')/table/tr"):
        for tr in trs:
            item = Car58Item()
            item['url'] = tr.xpath("td[@class='img']/a/@href").extract()
            item['tip'] = tr.xpath("td[@class='t']/a/font/text()").extract()
            item['name'] = tr.xpath("td[@class='t']/a[1]/text()").extract()
       #     item['description'] = trs.xpath("/td[@class='t']/a/text()").extract()
            item['size'] = tr.xpath("td[@class='t']/p").extract()
            item['region'] = tr.xpath("td[@class='tc']/a/text()").extract()
            item['amt'] = tr.xpath("td[@class='tc']/b/text()").extract()
            items.append(item)
        return items

scrapy内置的JSON导出是可以的。

相关问题
最新问题