Question

我正在尝试使用以下代码从this site中删除产品名称和价格：

class ProductSpider(scrapy.Spider):
name = 'product'

start_urls = ['https://www.bodyenfitshop.nl/']

def parse(self, response):
    # follow links to different categories
    for href in response.css('ol.nav-primary > li.category-node > a::attr(href)'):
        # A category page currently does not list all the items belonging to that category.
        # Add "?p=1" to get the list view
        href = href.extract() + "?p=1"
        print(href)
        yield SplashRequest(href, self.parse_category, args={
            'wait': 0.5
        })

def parse_category(self, response):
    for product in response.css('li.item'):
        new_name = product.css('div.product-name > a::text').extract_first().strip()
        # TODO: find better regex
        new_price = float(product.css('span.price::text').re('\d+,\d+')[0].replace(",", "."))
        new_url = product.css('div.product-name > a::attr(href)').extract_first().strip()
        yield Product(name=new_name, price=new_price, url=new_url)

我怀疑问题出在传递给SplashRequest的href中，但是当我打印出来时，他们都有完全合格的URL，如下所示：

https://www.bodyenfitshop.nl/duursport/?p=1

https://www.bodyenfitshop.nl/workouts/?p=1

https://www.bodyenfitshop.nl/aminozuren/?p=1

关于此错误的所有其他问题（例如this或this）都可以通过在其网址中添加“https”来解决。但我已经有了这些。所以我对造成这些问题的原因感到茫然。

这是我得到的错误之一（重复多次）

2017-05-31 22:51:07 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.

bodyenfitshop.nl/afslanken/?p=1 via https://www.bodyenfitshop.nl/afslanken/?p=1>
Traceback (most recent call last):
  File "c:\program files (x86)\python\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
    result = g.send(result)
  File "c:\program files (x86)\python\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
    response = yield method(request=request, spider=spider)
  File "c:\program files (x86)\python\lib\site-packages\scrapy_splash\middleware.py", line 358, in process_request
    priority=request.priority + self.rescheduling_priority_adjust
  File "c:\program files (x86)\python\lib\site-packages\scrapy\http\request\__init__.py", line 94, in replace
    return cls(*args, **kwargs)
  File "c:\program files (x86)\python\lib\site-packages\scrapy_splash\request.py", line 76, in __init__
    **kwargs)
  File "c:\program files (x86)\python\lib\site-packages\scrapy\http\request\__init__.py", line 25, in __init__
    self._set_url(url)
  File "c:\program files (x86)\python\lib\site-packages\scrapy\http\request\__init__.py", line 58, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: render.html

render.html是（我认为）Splash产生的默认名称。我认为我不能改变它。

非常感谢任何帮助或捅正确的方向！

Answer 1

在我的情况下，这是因为我忘记将https://放到SPLASH_URL

因此应为SPLASH_URL = https://yourdomain.com:8050

Scrapy + Splash：在请求url：render.html中缺少方案，但是url有方案

1 个答案: