我在文档中看到他们有CrawlSpider的示例代码:
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class MySpider(CrawlSpider):
name = 'example.com'
allowed_domains = ['example.com']
start_urls = ['http://www.example.com']
rules = (
# Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default).
Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
# Extract links matching 'item.php' and parse them with the spider's method parse_item
Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
)
def parse_item(self, response):
self.logger.info('Hi, this is an item page! %s', response.url)
item = scrapy.Item()
item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
return item
根据我的理解,这些步骤将会发生:
MySpider
)将从Scrapy Engine获取'http://www.example.com'
链接(位于start_url
列表中)的响应。然后LinkExtractor
将根据上面提供的两个规则从该响应中提取所有链接。LinkExtractor
(带回调)有3个链接('http://www.example.com/item1.php','http://www.example.com/item2.php','http://www.example.com/item3.php'
)和没有回调的第一个LinkExtractor
得到1个链接({{1 }})。对于上面找到的3个链接,将简单地调用指定的回调www.example.com/category1.php
。但是,对于那个链接(parse_item
)会发生什么,因为没有与之关联的回调?这两个www.example.com/category1.php
会再次在该链接上运行吗?这个假设是否正确?
答案 0 :(得分:0)
# Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default).
由于您的Rule
对象没有callback
参数,follow
参数设置为True
。
因此,在您的示例中,将抓取1个链接并从中提取链接,就像第一页所做的那样,这将继续,直到第一个规则不再提取链接或已经访问过所有链接。