Question

我正在使用scrapy下载图像，但无法正常工作。我在所需的文件夹中找到了URL，但没有图像。

这是我的item.py：

class Brand(scrapy.Item):
    name = scrapy.Field()
    url = scrapy.Field()
    brand_image = scrapy.Field()
    image_urls = scrapy.Field()
    images = scrapy.Field()
    pass

这是我的蜘蛛代码：

import scrapy
import json
from scraper.items import Brand


class QuotesSpider(scrapy.Spider):
    name = "brandDetails"

    def start_requests(self):
        with open('brands.json') as data_file:
            data_item = json.load(data_file)
        urls = list()
        for item in data_item:
            urls.append(item["url"])
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        item = Brand()
        name = response.css("div.th::text").extract_first()
        name = name.replace('Products of ', '')
        item['name'] = name
        item['url'] = response.url
        urls = response.css("div.productimage img::attr(src)").extract_first()
        urls = "http://ozhat-turkiye.com" + urls
        item['image_urls'] = urls
        yield item

这是设置代码：

BOT_NAME = 'scraper'
SPIDER_MODULES = ['scraper.spiders']
NEWSPIDER_MODULE = 'scraper.spiders'
IMAGE_STORE = 'C:/Users/SHAHRUKH/Desktop/AI'
ITEM_PIPELINES = {
  'scrapy.pipelines.images.ImagesPipeline': 1
}
DOWNLOAD_DELAY = 2

这是我的程序的输出：

 {'image_urls': 'http://ozhat-turkiye.com/get.aspx?id=1882267',
 'name': ' Camille Bauer',
 'url': 'http://ozhat-turkiye.com/en/camille-bauer/'}

Answer 1

您可以放下这些图像并使用urllib进行存储。

import urllib     

urllib.urlretrieve(img_url, filename)

例如代码检查here。

Scrapy无法从URL下载图像

1 个答案: