Question

我一直在尝试制作我的第一台履带式车，我已经完成了我需要的东西（获得1º商店和2º商店的运输信息和价格）但是有2个爬行器而不是1个，因为我在这里有一个很大的限制。

当有超过1家商店时，输出结果为：

In [1]: response.xpath('//li[@class="container list-display-box__list__container"]/div/div/div/div/div[@class="shipping"]/p//text()').extract()
Out[1]: 
[u'ENV\xcdO 3,95\u20ac ',
 u'ENV\xcdO GRATIS',
 u'ENV\xcdO GRATIS',
 u'ENV\xcdO 4,95\u20ac ']

只获取我正在使用的第二个结果：

In [2]: response.xpath('//li[@class="container list-display-box__list__container"]/div/div/div/div/div[@class="shipping"]/p//text()')[1].extract()
Out[2]: u'ENV\xcdO GRATIS'

但是当我没有第二个结果（只有一个商店）时，我得到了：

IndexError: list index out of range

即使其他项目有数据，抓取工具也会跳过整页...

经过多次尝试，我决定快速解决方案以获得结果，2个爬行器1用于第一个商店，另一个用于第二个商店但现在我只想在1个爬虫中干净。

一些帮助，提示或建议将不胜感激，这是我第一次尝试使用scrapy制作递归爬虫，有点像。

有代码：

# -*- coding: utf-8 -*-
import scrapy
from Guapalia.items import GuapaliaItem
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class GuapaliaSpider(CrawlSpider):
    name = "guapalia"
    allowed_domains = ["guapalia.com"]
    start_urls = (
        'https://www.guapalia.com/perfumes?page=1',
        'https://www.guapalia.com/maquillaje?page=1',
        'https://www.guapalia.com/cosmetica?page=1',
        'https://www.guapalia.com/linea-de-bano?page=1',
        'https://www.guapalia.com/parafarmacia?page=1',
        'https://www.guapalia.com/solares?page=1',
        'https://www.guapalia.com/regalos?page=1',
    )
    rules = (
        Rule(LinkExtractor(restrict_xpaths="//div[@class='js-pager']/a[contains(text(),'Siguientes')]"),follow=True),
        Rule(LinkExtractor(restrict_xpaths="//div[@class='list-display__item list-display__item--product']/div/a[@class='col-xs-10 col-sm-10 col-md-12 clickOnProduct']"),callback='parse_articles',follow=True),
    )
    def parse_articles(self, response):
        item = GuapaliaItem()
        articles_urls = response.url
        articles_first_shop = response.xpath('//div[@class="container-fluid list-display-box--best-deal"]/div/div/div/div[@class="retailer-logo autoimage-container"]/img/@title').extract()
        articles_first_shipping = response.xpath('//div[@class="container-fluid list-display-box--best-deal"]/div/div/div/div[@class="shipping"]/p//text()').extract()
        articles_second_shop = response.xpath('//li[@class="container list-display-box__list__container"]/div/div/div/div/div/img/@title')[1].extract()
        articles_second_shipping = response.xpath('//li[@class="container list-display-box__list__container"]/div/div/div/div/div[@class="shipping"]/p//text()')[1].extract()
        articles_name = response.xpath('//div[@id="ProductDetail"]/@data-description').extract()
        item['articles_urls'] = articles_urls
        item['articles_first_shop'] = articles_first_shop
        item['articles_first_shipping'] = articles_first_shipping
        item['articles_second_shop'] = articles_second_shop if articles_second_shop else 'N/A'
        item['articles_second_shipping'] = articles_second_shipping
        item['articles_name'] = articles_name
        yield item

当商店超过1时，具有正确格式的抓取工具的基本输出：

2017-09-21 09:53:11 [scrapy] DEBUG: Crawled (200) <GET https://www.guapalia.com/zen-edp-vaporizador-100-ml-75355> (referer: https://www.guapalia.com/perfumes?page=1)
2017-09-21 09:53:11 [scrapy] DEBUG: Scraped from <200 https://www.guapalia.com/zen-edp-vaporizador-100-ml-75355>
{'articles_first_shipping': [u'ENV\xcdO GRATIS'],
 'articles_first_shop': [u'DOUGLAS'],
 'articles_name': [u'ZEN edp vaporizador 100 ml'],
 'articles_second_shipping': u'ENV\xcdO 3,99\u20ac ',
 'articles_second_shop': u'BUYSVIP',
 'articles_urls': 'https://www.guapalia.com/zen-edp-vaporizador-100-ml-75355'}

问题是当第二家商店不存在时，因为我在第二家商店的代码

IndexError：列表索引超出范围

解决方案感谢@Tarun Lalwani

def parse_articles(self, response):
    item = GuapaliaItem()
    articles_urls = response.url
    articles_first_shop = response.xpath('//div[@class="container-fluid list-display-box--best-deal"]/div/div/div/div[@class="retailer-logo autoimage-container"]/img/@title').extract()
    articles_first_shipping = response.xpath('//div[@class="container-fluid list-display-box--best-deal"]/div/div/div/div[@class="shipping"]/p//text()').extract()
    articles_second_shop = response.xpath('//li[@class="container list-display-box__list__container"]/div/div/div/div/div/img/@title')
    articles_second_shipping = response.xpath('//li[@class="container list-display-box__list__container"]/div/div/div/div/div[@class="shipping"]/p//text()')
    articles_name = response.xpath('//div[@id="ProductDetail"]/@data-description').extract()
    if len(articles_second_shop) > 1:
        item['articles_second_shop'] = articles_second_shop[1].extract()
    else:
        item['articles_second_shop'] = 'Not Found'
    if len(articles_second_shipping) > 1:
        item['articles_second_shipping'] = articles_second_shipping[1].extract()
    else:
        item['articles_second_shipping'] = 'Not Found'
    item['articles_urls'] = articles_urls
    item['articles_first_shop'] = articles_first_shop
    item['articles_first_shipping'] = articles_first_shipping
    item['articles_name'] = articles_name
    yield item

Answer 1

您需要先在变量中获得结果。然后你可以根据它的长度做出决定

if (_formatDesc) {
    CFRelease(_formatDesc);
    _formatDesc = NULL;
}

Scrapy检测Xpath是否不存在

1 个答案: