Question

I created a pipeline to save each item on ElasticSearch. On this pipeline I check if item already exist to check if administrator override some field, to force a reindex (got this field and save/override it on new item)

class InsertItemOnElasticSearch(object):
    buffer = []

    @classmethod
    def from_crawler( cls, crawler ):
        # Init ES connection

    # Get uniq ID
    def got_id( self, item ):
        # Return uniq ID of item

    # Check if insert or update
    def check_item( self, item ):
        item_id = self.got_id( item )
        type = 'create'

        is_exist = self.es.search(...)

        if is_exist[ 'hits' ][ 'total' ] == 1:
            type = 'index'
            item_tmp = is_exist[ 'hits' ][ 'hits' ][0][ '_source' ]
            is_override_by_admin = item_tmp.get( 'is_override_by_admin', False )

            if is_override_by_admin:
                ...

                try:
                    my_field = item_tmp.get( 'my_field' )
                    if my_field:
                        item[ 'my_field' ] = my_field

                except:
                    pass

        return self.index_item( item, type, item_id )

    # Format indexation
    def index_item( self, item, op_type, item_id ):
        # Add es_action to buffer

    # Send buffer to ES
    def save_items( self ):
        helpers.bulk( self.es, self.buffer )

    # Process item send to pipelines
    def process_item( self, item, spider ):
        return self.check_item( item )

    # Send buffer when spider closed    
    def close_spider( self, spider ):
        if len( self.buffer ):
            self.save_items()

But, when a product exist and have my_field fill, script save same content on all next item, despite not having this field existed. So all my data is corrupted..

Someone know why ?

Answer 1

I fixed it. It's because on spider I call

for resp in self.gotVariantsListing(): # On product page
    ...
    yield resp
    ...

def gotVariantsListing(): # To get all variants of product
    ...
    for x in i:
        yield Request( MyURL, self.ajaxFunction, meta={ 'item': item }, ... )
    ...

def ajaxFunction(): # To got variant specific attributs
    item = response.meta[ 'item' ]
    ...
    yield item

So I change to meta={ 'item': item.copy() } and now it's works like expected. Thanks to remember me of .copy().. ;-)

We don't change crawler process so item reference is same for all ajaxFunction call. For that all changed on pipelines was readable on spider process

Why my pipeline return previous modified items?

1 个答案: