修改start_url参数时调用Spider时发生ValueError（'请求url中缺少方案：％s'％self._url）

时间：2019-01-23 22:15:12

标签： python scrapy web-crawler jupyter-notebook

我正在用刮刀工作，通过jupyter笔记本刮擦。

这是刮板的代码，保存在python文件中

import scrapy


from scrapy.selector import Selector
from scrapy.http import HtmlResponse

class GoogleApipoweredContentScraperSpider(scrapy.Spider):
    name = 'google_apipowered_content_scraper'

    start_urls = ['https://doc.scrapy.org/en/latest/topics/selectors.html#topics-selectors']

    def parse(self, response):
        texta = response.xpath("//text()").extract()
        texttitle = response.xpath('//title/text()').extract()
        yield {"Article_title" : texttitle, "Content_text" :texta}


</blink>

当您像这样正常调用和执行刮板时，刮板对初学者来说效果很好。 

<blink>
!scrapy runspider google_apipowered_content_scraper.py

</blink>

这就是问题，doc.scrapy起始URL仅作为执行示例，我实际上计划在调用时修改该变量（即发生错误的地方，这是我正在使用的代码：

!scrapy runspider google_apipowered_content_scraper.py -a "start_urls=[https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html]"

但是它不起作用，这里产生的错误是：

    019-01-23 20:21:43 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)
2019-01-23 20:21:43 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) - [GCC 7.2.0], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i  14 Aug 2018), cryptography 2.4.1, Platform Linux-4.15.0-43-generic-x86_64-with-debian-buster-sid
2019-01-23 20:21:43 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-01-23 20:21:43 [scrapy.core.engine] INFO: Spider opened
2019-01-23 20:21:43 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-01-23 20:21:43 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6027
2019-01-23 20:21:43 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/core/engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/spiders/__init__.py", line 83, in start_requests
    yield Request(url, dont_filter=True)
  File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/http/request/__init__.py", line 62, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: [
2019-01-23 20:21:43 [scrapy.core.engine] INFO: Closing spider (finished)
2019-01-23 20:21:43 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 1, 23, 19, 21, 43, 224181),
 'log_count/DEBUG': 1,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'memusage/max': 55255040,
 'memusage/startup': 55255040,
 'start_time': datetime.datetime(2019, 1, 23, 19, 21, 43, 219808)}
2019-01-23 20:21:43 [scrapy.core.engine] INFO: Spider closed (finished)

    </blink>

主要是错误似乎与此ValueError（“请求url中的缺少方案：％s'％self._url）

我曾尝试在调用参数时使用括号和不使用括号[]来编写start_url，我还尝试添加网址域，并在添加网址后添加了逗号，但未添加。在所有情况下，我都会得到相同的错误。

代码应该返回所有url内的文本，这些文本当然是杂乱无章的，但是所有的文本。正在处理此问题：'Python版本3.6.5 | Anaconda版本5.2.0，Jupyter Notebook 5.5.0

我完全不熟悉网页抓取功能，我不知道如何解决此问题，尽管我仍在尝试，但我一直在这一方面努力工作，我看到了很多帖子，但没有一个适合于一个案例像我的一样，我从jupyter笔记本中修改了start_url变量。他们中的大多数人都犯了不加[]的错误，但这显然不是我的情况。希望它能带来乐趣并让社区变得更聪明。提前致谢。

3 个答案:

答案 0 :(得分：1)

我变得非常有兴趣，并解决了您的问题））。 将此添加到您的代码中：

  1. 在类GoogleApipoweredContentScraperSpider中添加：

    my_start_page = ''

  2. 添加方法：

    def start_requests(self): if self.my_start_page is not None and self.my_start_page != '': self.my_start_page = self.my_start_page.split(',') for url in self.my_start_page: yield self.make_requests_from_url(url) else: for url in self.start_urls: yield self.make_requests_from_url(url)

测试！！！一切正常！

scrapy runspider google_apipowered_content_scraper.py -a my_start_page=https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html,https://www.yoursitenb2.com

如果您要将一个URL传输到开头，则只需删除split(',')并循环。只是：self.make_requests_from_url(self.my_start_page)

答案 1 :(得分：0)

!scrapy runspider google_apipowered_content_scraper.py -a "start_urls=['https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html']"

尝试

答案 2 :(得分：0)

很抱歉，但是在我看来，如果不更改Scrapy便无法执行此操作，因为命令行参数始终作为字符串传递给它。

def start_requests(self):
        cls = self.__class__
        if method_is_overridden(cls, Spider, 'make_requests_from_url'):
            warnings.warn(
                "Spider.make_requests_from_url method is deprecated; it "
                "won't be called in future Scrapy releases. Please "
                "override Spider.start_requests method instead (see %s.%s)." % (
                    cls.__module__, cls.__name__
                ),
            )
             for url in self.start_urls:
                yield self.make_requests_from_url(url)
        else:
            for url in self.start_urls:
                yield Request(url, dont_filter=True)

这是Scrapys代码的片段。

为了使您的论点理解Scrapy，您需要添加到Scrapy：

for url in self.start_urls:
   yield self.make_requests_from_url(url)

此：

for url in self.start_urls.strip('[]').split(','))
   yield self.make_requests_from_url(url)

并使用该参数运行Scrapy：

scrapy runspider google_apipowered_content_scraper.py -a start_urls=[https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html,https://www.another_url.com]

但我不确定是否没有更多的周期可以从数组中获取URL。需要找到。

这是拐杖）））

相关问题
最新问题