我是scrapy和python的新手。我写了一个蜘蛛,它与初始化的start_urls值一起工作正常。
如果我将一个文字放在Init as
中的代码中,它也可以正常工作{self.start_urls ='http://something.com'}
但是,当我从json文件中读取值并创建列表时,我得到了关于缺少%20的相同错误
我觉得我在scrapy或python中缺少一些明显的东西,因为我是一个nube。
class SiteFeedConstructor(CrawlSpider, FeedConstructor):
name = "Data_Feed"
start_urls = ['http://www.cnn.com/']
def __init__(self, *args, **kwargs):
FeedConstructor.__init__(self, **kwargs)
kwargs = {}
super(SiteFeedConstructor, self).__init__(*args, **kwargs)
self.name = str(self.config_json.get('name', 'Missing value'))
self.start_urls = str(self.config_json.get('start_urls', 'Missing value'))
self.start_urls = self.start_urls.split(",")
ERROR:
Traceback (most recent call last):
File "/usr/bin/scrapy", line 4, in <module>
execute()
File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 132, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 97, in _run_print_help
func(*a, **kw)
File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 139, in _run_command
cmd.run(args, opts)
File "/usr/lib/python2.7/dist-packages/scrapy/commands/runspider.py", line 64, in run
self.crawler.crawl(spider)
File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 42, in crawl
requests = spider.start_requests()
File "/usr/lib/python2.7/dist-packages/scrapy/spider.py", line 55, in start_requests
reqs.extend(arg_to_iter(self.make_requests_from_url(url)))
File "/usr/lib/python2.7/dist-packages/scrapy/spider.py", line 59, in make_requests_from_url
return Request(url, dont_filter=True)
File "/usr/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 26, in __init__
self._set_url(url)
File "/usr/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 61, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: Missing%20value
答案 0 :(得分:1)
而不是定义__init__()
覆盖start_requests()
方法:
当蜘蛛被打开时,这是Scrapy调用的方法 在没有指定特定URL时进行抓取。如果是特定的URL 如果指定,则使用make_requests_from_url()来创建 要求。这种方法也只在Scrapy中调用一次,所以就是这样 安全地将其作为发电机实施。
class SiteFeedConstructor(CrawlSpider, FeedConstructor):
name = "Data_Feed"
def start_requests(self):
self.name = str(self.config_json.get('name', 'Missing value'))
for url in str(self.config_json.get('start_urls', 'Missing value')).split(","):
yield self.make_requests_from_url(url)