Scrapy从管道获取返回值

时间:2017-01-06 17:41:58

标签: python django scrapy scrapy-spider scrapy-pipeline

我正在尝试从管道返回值。我正在使用yield生成器来生成项目。

这是我的代码。

def get_or_create(model):
    model_class = type(model)
    created = False

    try:
        obj = model_class.objects.get(product_company=model.product_company, barcode=model.barcode)
    except model_class.DoesNotExist:
        created = True
        obj = model  # DjangoItem created a model for us.
        obj.save()
    return (obj, created)


def update_model(destination, source, commit=True):
    pk = destination.pk
    source_dict = model_to_dict(source)
    for (key, value) in source_dict.items():
        setattr(destination, key, value)
    setattr(destination, 'pk', pk)
    if commit:
        destination.save()
    return destination
class ProductItemPipeline(object):
    def process_item(self, item, spider):

        if isinstance(item, ProductItem):
            item_model = item.instance
            model, created = get_or_create(item_model)
            item['cover_photo'] = item['files'][0]['path']
            if created:
                item.save()
                for image in item['files']:
                    imageItem = ProductImageItem(image=image['path'], product=model)
                    imageItem.save()
                for comment in item['comments']:
                    commentItem = CommentItem(comment=comment.comment, product=model)
                    commentItem.save()
            return model

这也是我的蜘蛛。

 item = ProductItem(name=name, price=price, barcode=barcode, file_urls=objectImages, product_url=response.url,product_company=company, comments = comments)
        product = yield item
        print type(product)
        print "yield product"

产品类型返回非类型

1 个答案:

答案 0 :(得分:0)

你不懂python { "items": [ #foreach($elem in $input.params('items').split(',')) { "ids": $elem.ids, "contents": $elem.contents }#if($foreach.hasNext),#end #end ], "QueryID": $input.params('QueryID'), "nR": $input.params('nR') } 。您应该使用yield之类的yield来返回一些功能结果。您可以在此article

中找到有关生成器的更多信息

所以你的代码看起来像

return