如何按蜘蛛类定义保持/导出特定顺序的字段项目,利用Scrapy中的项目管道

时间:2018-03-01 20:21:22

标签: python python-3.x scrapy scrapy-spider scrapy-pipeline

我有一个蜘蛛将数据导出到不同的CSV文件(根据蜘蛛类中定义的类定义的名称)。但是,我还希望在处理和导出到不同的CSV文件时,按特定顺序保留字段的顺序。

例如,这是我的items.py:

import scrapy

class first_class_def_Item(scrapy.Item):
    f1 = scrapy.Field() # f1 an arbitrary id used for both class definition items
    f2 = scrapy.Field()
    f3 = scrapy.Field()

class second_class_def_Item(scrapy.Item):
    f1 = scrapy.Field()
    f4 = scrapy.Field()
    f5 = scrapy.Field()
    f6 = scrapy.Field()

这是我的pipelines.py:

from scrapy.exporters import CsvItemExporter
from scrapy import signals
from pydispatch import dispatcher


def item_type(item):
    # The CSV file names are used (imported) from the scrapy spider.
    # For this example, I just want to keep "first_class_def.csv" without,
    # the "_item", as in "first_class_def_Item.csv" as defined in the main scrapy spider
    return type(item).__name__.replace('_Item','')

class SomeSitePipeline(object):
    # For simplicity, I'm using the same class def names as found in the,
    # main scrapy spider and as defined in the items.py
    SaveTypes = ['first_class_def','second_class_def']

    def __init__(self):
        dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
        dispatcher.connect(self.spider_closed, signal=signals.spider_closed)

    def spider_opened(self, spider):
        self.files = dict([ (name, open("/somefolder/"+name+'.csv','wb')) for name in self.SaveTypes ])
        self.exporters = dict([ (name,CsvItemExporter(self.files[name])) for name in self.SaveTypes ])
        [e.start_exporting() for e in self.exporters.values()]

    def spider_closed(self, spider):
        [e.finish_exporting() for e in self.exporters.values()]
        [f.close() for f in self.files.values()]

    def process_item(self, item, spider):
        typesItem = item_type(item)
        if typesItem in set(self.SaveTypes):
            self.exporters[typesItem].export_item(item)
        return item

这是我的spider.py:

import os
import scrapy
from itertools import zip_longest
from somesite.items import first_class_def_Item, second_class_def_Item
from csv import DictReader

path = os.path.join(os.path.expanduser('~'), 'user', 'somefolder', 'IDs.csv')

class SomeSiteSpider(scrapy.Spider):
    name = 'somesite'
    allowed_domains = ['somesite.com']
    start_urls = ['https://somesite.com/login.aspx']

    def parse(self, response):

        return scrapy.FormRequest.from_response(response,
                            formdata={'txtLogin$txtInput': 'User',
                                      'txtPassword$txtInput': 'pass',
                                      'btnLogin.x': '53',
                                      'btnLogin.y': '33'},
                            callback=self.Tables)

    def Tables(self, response):

        with open(path) as rows:

            for row in DictReader(rows):

                id=row["id"]

            yield scrapy.Request("https://somesite.com/page1.aspx",
                meta={'mid': mid,
                      'form_control': some_form_control},
                dont_filter = True,
                callback=self.first_class_def)

            yield scrapy.Request("https://somesite.com/page2.aspx",
                meta={'mid': mid,
                      'form_control': some_form_control},
                dont_filter = True,
                callback=self.second_class_def)

    def first_class_def(self, response):

        return scrapy.FormRequest.from_response(response,
                    formdata={'id': response.meta['id'],
                              'form_control': response.meta['some_form_control'],
                              'SearchControl$btnCreateReport': 'Create Report'},
                    meta={'id': response.meta['id']},
                    callback=self.scrap_page_1)

    def scrap_page_1(self, response):
        items = first_class_def_Item()

        field_1 = response.xpath('//*[@class="formatText"][1]/text()').extract()
        field_2 = response.xpath('//*[@class="formatCurrency"][1]/text()').extract()

        for a,b in zip(field_1,field_2):
            items['f1'] = response.meta['id']
            items['f2'] = a
            items['f3'] = b

            yield items

    def second_class_def(self, response):

        return scrapy.FormRequest.from_response(response,
                    formdata={'id': response.meta['id'],
                              'form_control': response.meta['some_form_control'],
                              'form_control_two': 'some_form_control_two',
                              'SearchControl$btnCreateReport': 'Create Report'},
                    meta={'id': response.meta['id']},
                    callback=self.scrap_page_2)

    def scrap_page_2(self, response):
        items = second_class_def_Item()

        field_1 = response.xpath('//*[@class="formatText"][1]/text()').extract()
        field_2 = response.xpath('//*[@class="formatCurrency"][1]/text()').extract()
        field_3 = response.xpath('//*[@class="formatText"][3]/text()').extract()

        for a,b,c in zip(field_1,field_2,field_3):
            items['f1'] = response.meta['id']
            items['f4'] = a
            items['f5'] = b
            items['f6'] = c

            yield items

当蜘蛛正在处理和导出数据时,我一直在寻找一种方法来保存CSV生成的文件中的字段" first_class_def.csv"和" second_class_def.csv",按照与items.py中相同的顺序导出:

F1,F2,F3

F1,F4,F5,F6

但是,每当我抓取蜘蛛时,CSV文件中的字段都是以随机顺序导出的:

f2,f1,f3和f5,f1,f4,f6

解决方案发布在下面!

2 个答案:

答案 0 :(得分:1)

不幸的是,由于实施了scrapy Item的方式,因此不会保留有关字段定义顺序的信息。

如果订单很重要,您可以做的最好的事情是将您想要的订单定义为单独的类变量,并在您的管道中使用它。将fields_to_export参数传递给CsvItemExporter可能是最简单的。

以下是您可以使用的基本想法:

# items.py
class Item1(scrapy.Item):
    fields_to_export = ['fi', 'f2']
    f1 = scrapy.Field()
    f2 = scrapy.Field()
# pipelines.py
from project.items import Item1


class SomeSitePipeline(object):
    save_types = {'item1': Item1}

    def spider_opened(self, spider):
        # (...)
        self.exporters = dict(
            (name, CsvItemExporter(self.files[name], fields_to_export=item_type.fields_to_export))
            for name, item_type in self.save_types.items()
        )
        # (...)

另外,我只是注意到你正在使用列表推导来解决副作用,这是一个坏主意,你应该只使用普通的循环。

答案 1 :(得分:0)

这是我特定问题的解决方案:根据scrapy spider项目的items.py中定义的items类定义组织导出字段。

所以在修补这个问题并实现@stranac建议摆脱列表理解之后,我提出了以下解决方案,允许将所有字段按顺序导出到相关的csv文件中:

File dbfile = new File("C:\Development\Maxmind GeoIP2 DB\GeoLite2-Country.mmdb");
DatabaseReader dbReader = new DatabaseReader.Builder(dbfile).build();

现在,一切都按照我原先的意图行事。