Question

通过提供的建议和大量的跟踪，我能够获得单页的抓取工作。现在，我试图对代码进行更改，以实现多个规则，但结果看起来不太好。以下是我正在尝试做的简要说明，

对于start_url = ttp：//sfbay.craigslist.org/ - 我使用parse_items_1来识别http://sfbay.craigslist.org/npo并解析相同内容以识别链接

在第2级，对于ttp：//sfbay.craigslist.org/npo中的链接，我需要使用parse_items_2来识别http://sfbay.craigslist.org/npo/index100.html之类的链接并解析它。

蜘蛛能够抓取（我可以看到显示），但链接不会被废弃。

2013-02-13 11:23:55+0530 [craigs] DEBUG: Crawled (200) <GET http://sfbay.craigslist.org/npo/index100.html> (referer: http://sfbay.craigslist.org/npo/)
('**parse_items_2:', [u'Development Associate'], [u'http://sfbay.craigslist.org/eby/npo/3610841951.html'])
('**parse_items_2:', [u'Resource Development Assistant'], [u'http://sfbay.craigslist.org/eby/npo/3610835088.html'])

但是报废时的链接和标题为空。

2013-02-13 11:23:55+0530 [craigs] DEBUG: Scraped from <200 http://sfbay.craigslist.org/npo/index100.html>
{'link': [], 'title': []}
2013-02-13 11:23:55+0530 [craigs] DEBUG: Scraped from <200 http://sfbay.craigslist.org/npo/index100.html>
{'link': [], 'title': []}

代码详情：

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from myspider.items import CraigslistSampleItem


class MySpider(CrawlSpider):
    name = "craigs"
    allowed_domains = ["sfbay.craigslist.org"]
    start_urls = ["http://sfbay.craigslist.org/"]

    rules = (
        Rule(SgmlLinkExtractor(allow=("index\d00\.html")), callback="parse_items_2", follow= True),
        Rule(SgmlLinkExtractor(allow=(r'sfbay.craigslist.org/npo')), callback="parse_items_1", follow= True),
        )

    def __init__(self, *a, **kw):
        super(MySpider, self).__init__(*a, **kw)
        self.items = []
        self.item = CraigslistSampleItem()

    def parse_items_1(self, response):
#       print response.url
        hxs = HtmlXPathSelector(response)
        titles = hxs.select("//div")
        for title in titles:
            self.item ["title"] = title.select("//li/a/text()").extract()
            self.item ["link"] = title.select("//li/a/@href").extract()
            print ('**parse-items_1:', self.item["title"])
            self.items.append(self.item)
        return self.items

    def parse_items_2(self, response):
#       print response.url
        hxs = HtmlXPathSelector(response)
        titles = hxs.select("//p")
        for title in titles:
            self.item ["title"] = title.select("a/text()").extract()
            self.item ["link"] = title.select("a/@href").extract()
            print ('**parse_items_2:', self.item["title"], self.item["link"])
            self.items.append(self.item)
        return self.items

非常感谢任何帮助！

谢谢。

Answer 1

在scrapy教程中，项目在回调中创建，然后返回以进一步向下传递，而不是绑定到蜘蛛类的实例。因此，删除init部分并重写一些回调代码似乎可以解决问题。

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from tutorial.items import CraigslistSampleItem

class MySpider(CrawlSpider):
    name = "craigs"
    allowed_domains = ["sfbay.craigslist.org"]
    start_urls = ["http://sfbay.craigslist.org/"]

    rules = (
        Rule(SgmlLinkExtractor(allow=("index\d00\.html")), callback="parse_items_2", follow= True),
        Rule(SgmlLinkExtractor(allow=(r'sfbay.craigslist.org/npo')), callback="parse_items_1", follow= True),
        )

    def parse_items_1(self, response):
        items = []
        hxs = HtmlXPathSelector(response)
        titles = hxs.select("//div")
        for title in titles:
            item = CraigslistSampleItem()
            item ["title"] = title.select("//li/a/text()").extract()
            item ["link"] = title.select("//li/a/@href").extract()
            print ('**parse-items_1:', item["title"])
            items.append(item)
        return items

    def parse_items_2(self, response):
        hxs = HtmlXPathSelector(response)
        titles = hxs.select("//p")
        items = []
        for title in titles:
            item = CraigslistSampleItem()
            item ["title"] = title.select("a/text()").extract()
            item ["link"] = title.select("a/@href").extract()
            print ('**parse_items_2:', item["title"], item["link"])
            items.append(item)
        return items

要进行测试，我将已抓取的项目转储到文件（scrapy crawl craigs -t json -o items.json）。我注意到每隔一段时间就有空条目和很多“使用条款”链接。这些表明你的提取xpaths可以收紧，但除此之外它似乎正在起作用。

Scrapy：爬行但不刮

1 个答案: