项目管道不适用于我使用scrapy从网站下载图像

时间:2018-09-29 05:08:44

标签: python-3.x selenium-webdriver scrapy pipeline scrapy-spider

下载图像对我不起作用,我也不知道我的程序出了什么问题

这是我的抓取程序的Setting.py

BOT_NAME = 'scraper'
SPIDER_MODULES = ['scraper.spiders']
NEWSPIDER_MODULE = 'scraper.spiders'
ITEM_PIPELINES = {'scraper.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = 'C:/Users/SHAHRUKH/scraper/brandImages/'

这是蜘蛛文件

import scrapy
import json


class Brand(scrapy.Item):
    name = scrapy.Field()
    url = scrapy.Field()
    brand_image = scrapy.Field()
    productsList = scrapy.Field()
    image_urls = scrapy.Field()
    images = scrapy.Field()
pass

class QuotesSpider(scrapy.Spider):
    name = "brandDetails"

    def start_requests(self):
        with open('brands.json') as data_file:
            data_item = json.load(data_file)
        urls = list()
        for item in data_item:
            urls.append(item["url"])
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        item = Brand()
        name = response.css("div.th::text").extract_first()
        name = name.replace('Products of ', '')
        item['name'] = name
        item['url'] = response.url
        url = response.css("div.productimage img::attr(src)").extract_first()
        url = "http://ozhat-turkiye.com" + url
        item['image_urls'] = url
        yield item

为什么在屈服项目上我的代码无法下载图像

0 个答案:

没有答案