Question

当我尝试从头下载图像时，我的数组为空，这似乎有些奇怪。我遵循了步骤，但是ImagePipeline似乎不适用于嵌套项目。但是，对我来说，在我的输出Json文件中保留这种清晰度非常重要。

这是我的代码：

    import json
import scrapy

from scrapy.loader import ItemLoader
from ccv_spiders.items import AuctionItem

class GlenMarchSpider(scrapy.Spider):
    name = 'auctions_results'

    def __init__(self, *args, **kwargs):
        with open('/home/bolgi/Workspace/Dev/python_workspace/ccv_spiders/ccv_spiders/json/input/db_urls_glenmarch_results_scrapy_reduced.json', encoding='utf-8') as data_file:
            self.data = json.load(data_file)

    def start_requests(self):
        for item in self.data:
            request = scrapy.Request(item['gm_url'], callback=self.parse)
            request.meta['item'] = item
            yield request

    def parse(self, response):
        item = response.meta['item']
        item['results'] = []

        for caritem in response.css("div.car-item-border"):
            data = AuctionItem()
            data["auction_house"] = caritem.css("div.auctionHouse::text").get().split("-", 1)[0].strip()
            data["auction_country"] = caritem.css("div.auctionHouse::text").get().rsplit(",", 1)[1].strip()
            data["auction_date"] = caritem.css("div.date::text").get().replace(",", "").strip()
            data["auction_url"] = caritem.css("div.view-auction a::attr(href)").get()
            data["img_urls"] = caritem.css("img.img-responsive::attr(src)").get()
            item['results'].append(data)

        yield item

项目文件：

import scrapy
from scrapy.loader.processors import MapCompose

class AuctionItem(scrapy.Item):
    auction_house= scrapy.Field()
    auction_country= scrapy.Field()
    auction_date= scrapy.Field()
    auction_url= scrapy.Field()
    image_urls= scrapy.Field()

    pass

和设置文件：

ITEM_PIPELINES = {
    'scrapy.pipelines.images.ImagesPipeline': 1,
}

IMAGES_STORE = 'images'

DOWNLOAD_TIMEOUT = 1200

我不明白如何使用Scrapy下载嵌套项目中的图片...

在Scrapy中的嵌套项目中使用ImagesPipeline

0 个答案: