我正在用刮刀工作，通过jupyter笔记本刮擦。
这是刮板的代码，保存在python文件中
import scrapy
from scrapy.selector import Selector
from scrapy.http import HtmlResponse
class GoogleApipoweredContentScraperSpider(scrapy.Spider):
name = 'google_apipowered_content_scraper'
start_urls = ['https://doc.scrapy.org/en/latest/topics/selectors.html#topics-selectors']
def parse(self, response):
texta = response.xpath("//text()").extract()
texttitle = response.xpath('//title/text()').extract()
yield {"Article_title" : texttitle, "Content_text" :texta}
</blink>
当您像这样正常调用和执行刮板时，刮板对初学者来说效果很好。
<blink>
!scrapy runspider google_apipowered_content_scraper.py
</blink>
这就是问题，doc.scrapy起始URL仅作为执行示例，我实际上计划在调用时修改该变量（即发生错误的地方，这是我正在使用的代码：
!scrapy runspider google_apipowered_content_scraper.py -a "start_urls=[https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html]"
但是它不起作用，这里产生的错误是：
019-01-23 20:21:43 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)
2019-01-23 20:21:43 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) - [GCC 7.2.0], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i 14 Aug 2018), cryptography 2.4.1, Platform Linux-4.15.0-43-generic-x86_64-with-debian-buster-sid
2019-01-23 20:21:43 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-01-23 20:21:43 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-01-23 20:21:43 [scrapy.core.engine] INFO: Spider opened
2019-01-23 20:21:43 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-01-23 20:21:43 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6027
2019-01-23 20:21:43 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/core/engine.py", line 127, in _next_request
request = next(slot.start_requests)
File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/spiders/__init__.py", line 83, in start_requests
yield Request(url, dont_filter=True)
File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
self._set_url(url)
File "/home/stefano/anaconda3/lib/python3.6/site-packages/scrapy/http/request/__init__.py", line 62, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: [
2019-01-23 20:21:43 [scrapy.core.engine] INFO: Closing spider (finished)
2019-01-23 20:21:43 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 1, 23, 19, 21, 43, 224181),
'log_count/DEBUG': 1,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'memusage/max': 55255040,
'memusage/startup': 55255040,
'start_time': datetime.datetime(2019, 1, 23, 19, 21, 43, 219808)}
2019-01-23 20:21:43 [scrapy.core.engine] INFO: Spider closed (finished)
</blink>
主要是错误似乎与此ValueError（“请求url中的缺少方案：％s'％self._url）
我曾尝试在调用参数时使用括号和不使用括号[]来编写start_url，我还尝试添加网址域，并在添加网址后添加了逗号，但未添加。在所有情况下，我都会得到相同的错误。
代码应该返回所有url内的文本，这些文本当然是杂乱无章的，但是所有的文本。正在处理此问题：'Python版本3.6.5 | Anaconda版本5.2.0，Jupyter Notebook 5.5.0
我完全不熟悉网页抓取功能，我不知道如何解决此问题，尽管我仍在尝试，但我一直在这一方面努力工作，我看到了很多帖子，但没有一个适合于一个案例像我的一样，我从jupyter笔记本中修改了start_url变量。他们中的大多数人都犯了不加[]的错误，但这显然不是我的情况。希望它能带来乐趣并让社区变得更聪明。提前致谢。
答案 0 :(得分：1)
我变得非常有兴趣，并解决了您的问题））。 将此添加到您的代码中：
在类GoogleApipoweredContentScraperSpider中添加：
my_start_page = ''
添加方法：
def start_requests(self):
if self.my_start_page is not None and self.my_start_page != '':
self.my_start_page = self.my_start_page.split(',')
for url in self.my_start_page:
yield self.make_requests_from_url(url)
else:
for url in self.start_urls:
yield self.make_requests_from_url(url)
测试！！！一切正常！
scrapy runspider google_apipowered_content_scraper.py -a my_start_page=https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html,https://www.yoursitenb2.com
如果您要将一个URL传输到开头，则只需删除split(',')并循环。只是：self.make_requests_from_url(self.my_start_page)
答案 1 :(得分：0)
!scrapy runspider google_apipowered_content_scraper.py -a "start_urls=['https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html']"
尝试
答案 2 :(得分：0)
很抱歉，但是在我看来，如果不更改Scrapy便无法执行此操作，因为命令行参数始终作为字符串传递给它。
def start_requests(self):
cls = self.__class__
if method_is_overridden(cls, Spider, 'make_requests_from_url'):
warnings.warn(
"Spider.make_requests_from_url method is deprecated; it "
"won't be called in future Scrapy releases. Please "
"override Spider.start_requests method instead (see %s.%s)." % (
cls.__module__, cls.__name__
),
)
for url in self.start_urls:
yield self.make_requests_from_url(url)
else:
for url in self.start_urls:
yield Request(url, dont_filter=True)
这是Scrapys代码的片段。
为了使您的论点理解Scrapy，您需要添加到Scrapy：
for url in self.start_urls:
yield self.make_requests_from_url(url)
此：
for url in self.start_urls.strip('[]').split(','))
yield self.make_requests_from_url(url)
并使用该参数运行Scrapy：
scrapy runspider google_apipowered_content_scraper.py -a start_urls=[https://elpais.com/internacional/2019/01/23/actualidad/1548249429_996363.html,https://www.another_url.com]
但我不确定是否没有更多的周期可以从数组中获取URL。需要找到。
这是拐杖）））