在Windows 7上全新安装了适用于Windows和Python 2.7的Miniconda 64位exe安装程序之后,通过该安装程序获得了Scrapy,以下是安装的内容:
这个最小的代码,来自" python scrapy_test.py" (使用Scrapy API):
#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-
import scrapy.spiders.crawl
import scrapy.crawler
import scrapy.utils.project
class MySpider(scrapy.spiders.crawl.CrawlSpider) :
name = "stackoverflow.com"
allowed_domains = ["stackoverflow.com"]
start_urls = ["http://stackoverflow.com/"]
download_delay = 1.5
def __init__(self, my_arg = None) :
print "def __init__"
self.my_arg = my_arg
print "self.my_arg"
print self.my_arg
def parse(self, response) :
pass
def main() :
my_arg = "Value"
process = scrapy.crawler.CrawlerProcess(scrapy.utils.project.get_project_settings())
process.crawl(MySpider(my_arg))
process.start()
if __name__ == "__main__" :
main()
给出了这个输出:
[scrapy] INFO: Scrapy 1.1.1 started (bot: scrapy_project)
[scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'scrapy_project.spiders', 'SPIDER_MODULES': ['scrapy_project.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'scrapy_project'}
def __init__
self.my_arg
Value
[scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
def __init__
self.my_arg
None
[...]
注意init方法是如何运行两次的,以及第二次运行后存储的参数如何变为None,这不是我想要的。这应该发生吗?
如果我改变:
def __init__(self, my_arg = None) :
为:
def __init__(self, my_arg) :
输出是:
[...]
Unhandled error in Deferred:
[twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "scrapy_test.py", line 28, in main
process.crawl(MySpider(my_arg))
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 163, in crawl
return self._crawl(crawler, *args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 167, in _crawl
d = crawler.crawl(*args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\twisted\internet\defer.py", line 1331, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "C:\Users\XYZ\Miniconda2\lib\site-packages\twisted\internet\defer.py", line 1185, in _inlineCallbacks
result = g.send(result)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 90, in crawl
six.reraise(*exc_info)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 71, in crawl
self.spider = self._create_spider(*args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 94, in _create_spider
return self.spidercls.from_crawler(self, *args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\crawl.py", line 96, in from_crawler
spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\__init__.py", line 50, in from_crawler
spider = cls(*args, **kwargs)
exceptions.TypeError: __init__() takes exactly 2 arguments (1 given)
[twisted] CRITICAL:
Traceback (most recent call last):
File "C:\Users\XYZ\Miniconda2\lib\site-packages\twisted\internet\defer.py", line 1185, in _inlineCallbacks
result = g.send(result)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 90, in crawl
six.reraise(*exc_info)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 71, in crawl
self.spider = self._create_spider(*args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 94, in _create_spider
return self.spidercls.from_crawler(self, *args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\crawl.py", line 96, in from_crawler
spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs)
File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\__init__.py", line 50, in from_crawler
spider = cls(*args, **kwargs)
TypeError: __init__() takes exactly 2 arguments (1 given)
不知道如何解决这个问题。有什么想法吗?
答案 0 :(得分:5)
以下是scrapy.crawler.CrawlerProcess.crawl()
的方法定义:
crawl(crawler_or_spidercls, *args, **kwargs)
- crawler_or_spidercls (
Crawler
实例,Spider
子类或字符串) - 已创建的抓取工具,或蜘蛛类或蜘蛛的名称 在项目内部创建它- args ( list ) - 初始化蜘蛛的参数
- kwargs ( dict ) - 初始化蜘蛛的关键字参数
这意味着您应该将Spider
的名称与初始化kwargs
所需的Spider
分开传递,如下所示:
process.crawl(MySpider, my_arg = 'Value')