Question

我使用Scrapy抓取网站上的项目。有很多信息，包括图像网址。您能帮我弄清楚如何从这些网址（data["image_urls"]）中提取图片。

我知道我必须扩展媒体管道，因为Scrapy不管理嵌套的url，但是我在过程中迷路了。

import json
import scrapy
import re
import pkgutil

from scrapy.loader import ItemLoader
from auctions_results.items import AuctionItem
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem

from datetime import datetime

class Spider(scrapy.Spider):

    name = 'results'

    def __init__(self, *args, **kwargs):
        data_file = pkgutil.get_data(
            "auctions_results", "json/input/scrape_demo_db.json")
        self.data = json.loads(data_file)

    def start_requests(self):
        for item in self.data:
            request = scrapy.Request(item['gm_url'], callback=self.parse)
            request.meta['item'] = item
            yield request

    def parse(self, response):
        item = response.meta['item']
        item['results'] = []

        for caritem in response.css("div.car-item-border"):
            data = AuctionItem()

            data["marque"] = caritem.css("div.make::text").extract_first().strip().split(" ", 2)[1]

            data["auction_house"] = caritem.css("div.auctionHouse::text").extract_first().split("-", 1)[0].strip()
            data["auction_country"] = caritem.css("div.auctionHouse::text").extract_first().rsplit(",", 1)[1].strip()
            data["auction_date"] = caritem.css("div.date::text").extract_first().replace(",", "").strip()

            data["image_urls"] = caritem.css("div.view-auction a img::attr(src)").extract_first()
            item['results'].append(data)

        yield item

当我用JSON编码结果时，它看起来像这样：

[{
"gm_url": "url",
"results": [{
    "marque": "ferrari",
    "auction_house": "auction",
    "auction_country": "japan",
    "auction_date": "2019",
    "image_urls": "imgurl"
},
{
    "marque": "porsche",
    "auction_house": "auction2",
    "auction_country": "gb",
    "auction_date": "2018",
    "image_urls": "imgurl2"
}]

}，...。 ]

我将图像项添加到items.py中，并在settings.py中激活ITEM_PIPELINE和IMAGES_STORE。

Answer 1

为此，您可以覆盖ImagesPipeline的get_media_requests方法：

class DownloadImagesPipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
        for result in item['results']:
            image_url = result['image_urls']
            request = Request(url=image_url,
                              headers=headers)
            yield request

在设置中，您应停用ImagesPipeline并将其替换为此“ DownloadImagesPipeline”。

如何从之前抓取的网址（嵌套的网址）中抓取图片

1 个答案: