Question

我正在测试刮擦和飞溅。我有一个蜘蛛：

class MySpider(scrapy.Spider):
    # start_urls = ["http://yahoo.com"]
    name = 'mytest'

    def __init__(self, state='CA', city='San_Francisco', *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.state = state
        self.city = city
        self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]
        # self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')


    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)

如果我放弃了init函数并取消对strt-urls行的注释，则Spider可以工作。逐步解决问题出现在：

self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]

回溯：

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
  File "  \lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "....\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 245, in spider_opened
    spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "....\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 165, in process_start_requests
    spider.state.setdefault(self.local_values_key, {})  # fingerprint => value dict

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
  File "  \lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "....\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 245, in spider_opened
    spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "....\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 165, in process_start_requests
    spider.state.setdefault(self.local_values_key, {})  # fingerprint => value dict
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
 'log_count/ERROR': 2,
 'log_count/INFO': 9,
 'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
 'log_count/ERROR': 2,
 'log_count/INFO': 9,
 'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)

Answer 1

问题在于当您查看易用的中间件代码时，您的参数名称为state

https://github.com/scrapy-plugins/scrapy-splash/blob/e40ca4f3b367ab463273bee1357d3edfe0601f0d/scrapy_splash/middleware.py#L245

state具有特殊含义，并且使用dict。因此，您需要做的就是重命名参数

class MySpider(scrapy.Spider):
    # start_urls = ["http://yahoo.com"]
    name = 'mytest'

    def __init__(self, STATE='CA', CITY='San_Francisco', *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.state = state
        self.city = city
        self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=STATE, city=CITY)]
        # self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')


    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)

Scrapy：AttributeError：'str'对象没有属性'setdefault'

1 个答案: