Question

我尝试从主要的解析函数中调用getNext（）函数，但是它永远不会被调用。

class BlogSpider(scrapy.Spider):
      # User agent.
      name = 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19'
      start_urls = ['http://www.tricksforums.org/best-free-movie-streaming-sites-to/']

      def getNext(self):
        print("Getting next ... ")
        # Check if next link in DB is valid and crawl.
        try:
          nextUrl = myDb.getNextUrl()
          urllib.urlopen(nextUrl).getcode()
          yield scrapy.Request(nextUrl['link'])
        except IOError as e:
          print("Server can't be reached", e.code)
          yield self.getNext()

      def parse(self, response):
        print("Parsing link: ", response.url)
        # Get all urls for futher crawling.
        all_links = hxs.xpath('*//a/@href').extract()
        for link in all_links:
          if validators.url(link) and not myDb.existUrl(link) and not myDb.visited(link):
            myDb.addUrl(link)
        print("Getting next?")
        yield self.getNext()

我之前尝试过，没有屈服。问题是什么？这个收益应该是什么？：）

Answer 1

您正在尝试生成一个生成器，但意味着从生成器生成。

如果您使用的是Python 3.3+，则可以使用yield from：

yield from self.getNext()

或者，只需执行return self.getNext()。

Python Scrapy函数调用

1 个答案: