Question

我问过这样的问题Scrapy can't get data。但是当我使用另一只蜘蛛时我遇到了一个新问题。我已经注意了xpath，但是这个程序似乎有同样的错误。

这是我的蜘蛛代码：

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import Selector
from scrapy import Item, Field
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from DB_Connection import DB_Con

class UniParc(Item):
    database = Field()
    identifier = Field()
    version = Field()
    organism = Field()
    first_seen = Field()
    last_seen = Field()
    active = Field()
    source = Field()

class UniParcSpider(CrawlSpider):
    name = "UniParc"
    allowed_domains = ["uniprot.org"]
    start_urls = ["http://www.uniprot.org/uniparc/?query=rna&offset=25&sort=score&columns=id%2corganisms%2ckb%2cfirst-seen%2clast-seen%2clength"]

    rules = (
        Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//*[@id="results"]/tr/td[2]/a',)), callback="parse_items", follow = True),
    )

    def parse_items(self, response):
        hxs = Selector(response)
        sites = hxs.xpath('//*[@id="results"]/tr')
        db = DB_Con()
        collection = db.getcollection(self.term)
        for site in sites:
            item = UniParc()
            item["database"] = map(unicode.strip, site.xpath("td[1]/text()").extract())
            item["identifier"] = map(unicode.strip, site.xpath("td[2]/a/text()").extract())
            item["version"] = map(unicode.strip, site.xpath("td[3]/text()").extract())
            item["organism"] = map(unicode.strip, site.xpath("td[4]/a/text()").extract())
            item["first_seen"] = map(unicode.strip, site.xpath("td[5]/text()").extract())
            item["last_seen"] = map(unicode.strip, site.xpath("td[6]/text()").extract())
            item["active"] = map(unicode.strip, site.xpath("td[7]/text()").extract())
            item['source'] = self.name
            collection.update({"identifier": item['identifier']}, dict(item), upsert=True)
            yield item

我使用rules提取我想要关注的链接并从中获取数据。但似乎没有从start_url获得网址。

这是日志：

2016-05-28 22:28:54 [scrapy] INFO: Enabled item pipelines: 
2016-05-28 22:28:54 [scrapy] INFO: Spider opened
2016-05-28 22:28:54 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-05-28 22:28:54 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-05-28 22:28:55 [scrapy] DEBUG: Crawled (200) <GET http://www.uniprot.org/uniparc/?query=rna&offset=25&sort=score&columns=id%2corganisms%2ckb%2cfirst-seen%2clast-seen%2clength> (referer: None)
2016-05-28 22:28:55 [scrapy] INFO: Closing spider (finished)
2016-05-28 22:28:55 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 314,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 12263,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 5, 28, 14, 28, 55, 638618),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2016, 5, 28, 14, 28, 54, 645490)}

所以有人能说出我的代码有什么问题吗？我的xpath有什么问题吗？但我已经多次检查过了。

Answer 1

为了修复以下链接步骤，只需修复XPath表达式，替换：

//*[@id="results"]/tr/td[2]/a

使用：

//*[@id="results"]//tr/td[2]/a

而且，作为旁注，您不应该直接在蜘蛛中将提取的项目插入数据库。为此，Scrapy提供pipelines。如果是MongoDB，请查看scrapy-mongodb。另见：

Web Scraping With Scrapy and MongoDB

在跟踪链接时，Scrapy无法获取数据

1 个答案: