出于某种原因，我的Scrapy蜘蛛抛出了一个错误，这与输出混淆，即使我使用条件语句来防止这种情况。
脚本
import scrapy
class AmazonSpider(scrapy.Spider):
name = 'amazon'
def start_requests(self):
pages = ['https://www.amazon.ca/electronics-deals-electronics-sale-tv-sale/b/ref=sv_ce_0?ie=UTF8&node=2055586011']
yield scrapy.Request(url=pages[0], callback=self.parse)
def parse(self, response):
if response.status != 404:
cont_path = '//div[@class="a-row a-spacing-none"]/span[@class="a-size-small a-color-secondary a-text-strike"]/ancestor::div[@class="s-item-container"]'
cont_resp = response.xpath(cont_path).extract()
for idx in range(1, len(cont_resp)+1):
thrd_path = '(//div[@class="a-row a-spacing-none"]/span[@class="a-size-small a-color-secondary a-text-strike"]/ancestor::div[@class="s-item-container"])[' + str(idx) + ']/div[@class="a-row a-spacing-mini"][2]/div[4]/a/span[2]/text()'
thrd_resp = response.xpath(thrd_path).extract_first()
### i have also tried:
# != None
# and hasattr(thrd_resp, 'split')
if thrd_resp is not None:
obj['thrd_resp'] = float(pric_resp.split()[1])
###
yield obj
日志
AttributeError: 'NoneType' object has no attribute 'split'
scrapy runspider amazon.py -o deals.json
2018-06-12 14:35:33 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-06-12 14:35:33 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.4.0, Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Windows-10-10.0.17134
2018-06-12 14:35:33 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True, 'FEED_FORMAT': 'json', 'FEED_URI': 'deals.json', 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'}
2018-06-12 14:35:33 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2018-06-12 14:35:33 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-06-12 14:35:33 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-06-12 14:35:33 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-06-12 14:35:33 [scrapy.core.engine] INFO: Spider opened
2018-06-12 14:35:33 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-06-12 14:35:33 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-06-12 14:35:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.ca/electronics-deals-electronics-sale-tv-sale/b/ref=sv_ce_0?ie=UTF8&node=2055586011> (referer: None)
2018-06-12 14:35:34 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.ca/electronics-deals-electronics-sale-tv-sale/b/ref=sv_ce_0?ie=UTF8&node=2055586011>
{'img': u'https://images-na.ssl-images-amazon.com/images/I/41YquQ15BdL._AA160_.jpg', 'thrd_resp': 90.35, 'pdr': u'Monster', 'len': 6, 'idx': 1, 'ttl': u'Monster BackFloat High Definition Bluetooth Wireless Waterproof Floating Speaker, Black/Blue'}
2018-06-12 14:35:34 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.amazon.ca/electronics-deals-electronics-sale-tv-sale/b/ref=sv_ce_0?ie=UTF8&node=2055586011> (referer: None)
Traceback (most recent call last):
File "c:\python27\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 30, in process_spider_output
for x in result:
File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "c:\python27\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "C:\Users\Anthony\Desktop\spiders\amazon\amazon.py", line 54, in parse
obj['thrd_resp'] = float(pric_resp.split()[1])
AttributeError: 'NoneType' object has no attribute 'split'
2018-06-12 14:35:34 [scrapy.core.engine] INFO: Closing spider (finished)
2018-06-12 14:35:34 [scrapy.extensions.feedexport] INFO: Stored json feed (1 items) in: deals.json
2018-06-12 14:35:34 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 363,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 127767,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 6, 12, 21, 35, 34, 903000),
'item_scraped_count': 1,
'log_count/DEBUG': 3,
'log_count/ERROR': 1,
'log_count/INFO': 8,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'spider_exceptions/AttributeError': 1,
'start_time': datetime.datetime(2018, 6, 12, 21, 35, 33, 652000)}
2018-06-12 14:35:34 [scrapy.core.engine] INFO: Spider closed (finished)
答案 0 :(得分：5)
if thrd_resp is not None:
obj['thrd_resp'] = float(pric_resp.split()[1])
您要分割pric_resp，但要检查thrd_resp。即使thrd_resp不是None，但这并不意味着pric_resp不是无，可以拆分。