Question

以下是代码。基本上，我正在抓电影信息。来自IMDB.com。但不知何故，请求并没有废弃URL，这是在对象＆＃34; addr＆＃34;中。＆＃34;打印＆＃34;我放入parse_item2只是没有显示出来。

这让我发疯。我花了几个小时。任何有经验帮助的人都可以吗？非常感谢你。

 # code for the spider

    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
    from scrapy.selector import Selector
    from scrapy.http import Request, Response
    from beta.items import BetaItem
    import urllib2

    class AlphaSpider(CrawlSpider):

    name = 'alpha'
    allowed_domains = ['amazon.com','imdb.com']
    start_urls = ['http://www.imdb.com/search/title?at=0&sort=boxoffice_gross_us&title_type=feature&year=2005,2005']
    rules = (Rule(SgmlLinkExtractor(restrict_xpaths=('//td/a',), allow=('/title/')), callback='parse_item1'),
    )

    def parse_item1(self, response):
        sel = Selector(response)
        item = BetaItem()

        idb = sel.xpath('//link[@rel="canonical"]/@href').extract()
        idb = idb[0].split('/')[-2]
        item['idb'] = idb

        title = sel.xpath('//h1[@class="header"]/span[@class="itemprop"]/text()').extract()
        item['title'] = title

        addr = 'http://www.imdb.com/title/' + idb + '/business'
        request = Request(addr, callback=self.parse_item2)
        request.meta['item'] = item
        return request

    def parse_item2(self, response):
        print 'I am here'
        item = response.meta['item']
        sel = Selector(response)
        # BLA BLA BLA
        return item

Answer 1

问题的原因确实如同Blender在上面的评论中所说的那样。为某些特定请求抓取内容需要相当长的时间。

Scrapy不遵循请求网址

1 个答案: