Question

我正在尝试运行一个Webspider，它可以获取特定URL的所有URL。现在，当我知道还有成千上万个网址时，它将返回约64个网址。有人知道为什么要早完成吗？

class MySpider(BaseSpider):
    custom_settings = {
        'AUTOTHROTTLE_ENABLED': True,
        'DOWNLOAD_DELAY': 1.5
    }

    name = 'www.shopgoodwill.com'
    allowed_domains = ['www.shopgoodwill.com']
    start_urls = [
        'https://www.shopgoodwill.com'
    ]

    def __init__(self, alexa_site_id, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.alexa_site_id = alexa_site_id

    def parse(self, response):
        le = LinkExtractor()
        for link in le.extract_links(response):
            yield Request(link.url, callback=self.parse_item)

以下是结果，我注意到说的是request_depth_max：1，但是我的设置中有DEPTH_LIMIT = 0

2019-02-19 23:31:03 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 14739,
 'downloader/request_count': 32,
 'downloader/request_method_count/GET': 32,
 'downloader/response_bytes': 336986,
 'downloader/response_count': 32,
 'downloader/response_status_count/200': 23,
 'downloader/response_status_count/302': 9,
 'dupefilter/filtered': 11,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 2, 19, 23, 31, 3, 824302),
 'log_count/DEBUG': 38,
 'log_count/INFO': 22,
 'memusage/max': 108908544,
 'memusage/startup': 108908544,
 'offsite/domains': 5,
 'offsite/filtered': 5,
 'request_depth_max': 1,
 'response_received_count': 23,
 'scheduler/dequeued': 32,
 'scheduler/dequeued/memory': 32,
 'scheduler/enqueued': 32,
 'scheduler/enqueued/memory': 32,
 'start_time': datetime.datetime(2019, 2, 19, 23, 30, 4, 918201)}
2019-02-19 23:31:03 [scrapy.core.engine] INFO: Spider closed (finished)
Closing spider (finished)
Dumping Scrapy stats:
{'downloader/request_bytes': 14739,
 'downloader/request_count': 32,
 'downloader/request_method_count/GET': 32,
 'downloader/response_bytes': 336986,
 'downloader/response_count': 32,
 'downloader/response_status_count/200': 23,
 'downloader/response_status_count/302': 9,
 'dupefilter/filtered': 11,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 2, 19, 23, 31, 3, 824302),
 'log_count/DEBUG': 38,
 'log_count/INFO': 22,
 'memusage/max': 108908544,
 'memusage/startup': 108908544,
 'offsite/domains': 5,
 'offsite/filtered': 5,
 'request_depth_max': 1,
 'response_received_count': 23,
 'scheduler/dequeued': 32,
 'scheduler/dequeued/memory': 32,
 'scheduler/enqueued': 32,
 'scheduler/enqueued/memory': 32,
 'start_time': datetime.datetime(2019, 2, 19, 23, 30, 4, 918201)}
Spider closed (finished)

Answer 1

根据我们对问题的评论，您也需要在parse_item()中提取链接。如果您仅提取parse()，则将不跟随后续链接。

尽早完成，没有获得所有链接

1 个答案: