Question

尝试使用带Scrapy的蜘蛛下载图像时出现以下错误。

File "C:\Python27\lib\site-packages\scrapy\http\request\__init__.py",
line 61, in _set_url
            raise ValueError('Missing scheme in request url: %s' % self._url)
        exceptions.ValueError: Missing scheme in request url: h

尽管我能理解它，看起来我在某个地方错过了一个“h”？但我不能为我的生活看到哪里。如果我不想下载图像，一切正常。但是，一旦我将相应的代码添加到下面的四个文件中，我就无法正常工作。任何人都可以帮我理解这个错误吗？

items.py

import scrapy

class ProductItem(scrapy.Item):
    model = scrapy.Field()
    shortdesc = scrapy.Field()
    desc = scrapy.Field()
    series = scrapy.Field()
    imageorig = scrapy.Field()
    image_urls = scrapy.Field()
    images = scrapy.Field()

settings.py

BOT_NAME = 'allenheath'

SPIDER_MODULES = ['allenheath.spiders']
NEWSPIDER_MODULE = 'allenheath.spiders'

ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}

IMAGES_STORE = 'c:/allenheath/images'

pipelines.py

class AllenheathPipeline(object):
    def process_item(self, item, spider):
        return item

import scrapy
from scrapy.contrib.pipeline.images import ImagesPipeline
from scrapy.exceptions import DropItem

class MyImagesPipeline(ImagesPipeline):

    def get_media_requests(self, item, info):
        for image_url in item['image_urls']:
            yield scrapy.Request(image_url)

    def item_completed(self, results, item, info):
        image_paths = [x['path'] for ok, x in results if ok]
        if not image_paths:
            raise DropItem("Item contains no images")
        item['image_paths'] = image_paths
        return item

products.py （我的蜘蛛）

import scrapy

from allenheath.items import ProductItem
from scrapy.selector import Selector
from scrapy.http import HtmlResponse

class productsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["http://www.allen-heath.com/"]
    start_urls = [
        "http://www.allen-heath.com/ahproducts/ilive-80/",
        "http://www.allen-heath.com/ahproducts/ilive-112/"
    ]

    def parse(self, response):
        for sel in response.xpath('/html'):
            item = ProductItem()
            item['model'] = sel.css('#prodsingleouter > div > div > h2::text').extract()
            item['shortdesc'] = sel.css('#prodsingleouter > div > div > h3::text').extract()
            item['desc'] = sel.css('#tab1 #productcontent').extract()
            item['series'] = sel.css('#pagestrip > div > div > a:nth-child(3)::text').extract()
            item['imageorig'] = sel.css('#prodsingleouter > div > div > h2::text').extract()
            item['image_urls'] = sel.css('#tab1 #productcontent img').extract()[0]
            item['image_urls'] = 'http://www.allen-heath.com' + item['image_urls']
            yield item

非常感谢任何帮助。

Answer 1

问题在于：

expected= 12.25, actual= 12.75 
expected= 4093.25, actual= 4094.75

在这里：

def get_media_requests(self, item, info):
    for image_url in item['image_urls']:
        yield scrapy.Request(image_url)

您正在提取此字段并获取第一个元素。这意味着一旦你在管道中迭代它，你实际上是迭代URL中的字符，以item['image_urls'] = sel.css('#tab1 #productcontent img').extract()[0]开头 - 解释你看到的错误信息，一旦第一个字母试图成为处理：

http

从该行中删除Missing scheme in request url: h。当你进入它时，获取图像的[0]，而不是整个元素：

src

之后，如果图像网址是相对的，您还应该更新下一行，将其转换为绝对值：

item['image_urls'] = sel.css('#tab1 #productcontent img').xpath('./@src').extract()

但如果import urlparse # put this at the top of the script item['image_urls'] = [urlparse.urljoin(response.url, url) for url in item['image_urls']]中的图片网址实际上是绝对的，那么您不需要这最后一部分，所以只需删除它。

使用Scrapy下载图像时出现问题

1 个答案: