Scrapy不会抓取

Question

    # -*- coding: utf-8 -*-
    import scrapy

    class ProvasSpider(scrapy.Spider):
        name = 'provas'
        allowed_domains = ['folhadirigida.com.br']
        start_urls = ['https://folhadirigida.com.br/']

        def parse(self, response): #criando pagina
            page = response.url.split ("/")[-3]
            filename = '%s.html' % page
            with open(filename, 'wb') as f:
                f.write(response.body)

运行此程序时，对this page进行爬网时，会得到：。 例如，如果我在this page上运行相同的程序，则会得到此页面的精确副本。为什么它不适用于首页？

Answer 1

由于您的HTML页面中缺少<base href="https://folhadirigida.com.br/">，对您来说看起来不太好。

import scrapy

class ProvasSpider(scrapy.Spider):
    name = 'provas'
    allowed_domains = ['folhadirigida.com.br']
    start_urls = ['https://folhadirigida.com.br/']

    def parse(self, response): #criando pagina
        page = response.url.split ("/")[-2]
        filename = '%s.html' % page
        with open(filename, 'wb') as f:
            ref_body = response.body[:42] + b'<base href="https://folhadirigida.com.br/">'\
                       + response.body[42:]
            f.write(ref_body)

像下面的代码一样将其添加到HTML正文中，可使页面看起来不错。