Question

我几个月来一直在研究这种蜘蛛，并且遇到了同样的问题 - 任何人都可以帮助我吗？

在上述网站（见下文）中，除了粗体“型号名称”之外，所有仪器数据都被删除。这真是令人生气，而且我无所适从。

import re
import json
from urlparse import urlparse


from scrapy.selector import Selector
try:
    from scrapy.spider import Spider
except:
    from scrapy.spider import BaseSpider as Spider
from scrapy.utils.response import get_base_url
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.selector import HtmlXPathSelector
from database.items import databaseItem

from scrapy.log import *

class CommonSpider(CrawlSpider):
    name = 'brands.py'
    allowed_domains = ['usedprice.com']
    start_urls = ['http://www.usedprice.com/items/guitars-musical-instruments/index.html']

    rules = (

        Rule(LinkExtractor(allow=( )), callback='parse_item'),
    )


    def parse_item(self, response):
        hxs = HtmlXPathSelector(response)
        item = databaseItem()
        datao = datao = hxs.xpath('//tr[@class="oddItemColor baseText"]')
        datae = datae = hxs.xpath('//tr[@class="evenItemColor baseText"]')
        tmpNextPage = hxs.xpath('//div[@class="baseText blue"]/span[@id="pnLink"]/a/@href').extract()
        for attr in datao:
            *modelInfo = attr.xpath('.//b/text()').extract()*
            instrInfo = attr.xpathxpath('.//td//text()').extract()
            item['modelInfo'].append = modelInfo
            item['instrInfo'].append = instrInfo
            return databaseItem(modelInfo = modelInfo[1:], instrInfo = instrInfo[2:])
        for attr in datae:
            *modelInfo = attr.xpath('.//b/text()').extract()*
            instrInfo = attr.xpath('.//td//text()').extract()
            item['modelInfo'].append = modelInfo
            item['instrInfo'].append = instrInfo
            return databaseItem(modelInfo = modelInfo[1:], instrInfo = instrInfo[2:])

Answer 1

你应该：

使用精确数组索引（而不是切片）来获取modelInfo字段
- [0][1:] == []
yield每个databaseItem而不是返回它
- https://stackoverflow.com/a/21672898
添加一些逻辑来检测粗体文本是否实际上是您想要的
- 感兴趣的表格中的每一行都有＆＃34;描述＆＃34;粗体，没有＆＃34;列＆＃34;对于＆＃34; Type＆＃34;信息

（我没有运行你的代码，但我确实看过http://www.usedprice.com/items/guitars-musical-instruments/a-basses/index.html）

Scrapy不会刮掉粗体文字

1 个答案: