Question

Scrapy文档版本1.0.3中有一个示例，在第7行中，当链接是相对的时，使用 urljoin 方法。当链接是绝对的时，我该怎么办？

示例代码：

import scrapy


class StackOverflowSpider(scrapy.Spider):
    name = 'stackoverflow'
    start_urls = ['http://stackoverflow.com/questions?sort=votes']

    def parse(self, response):
        for href in response.css('.question-summary h3 a::attr(href)'):
            full_url = response.urljoin(href.extract())
            yield scrapy.Request(full_url, callback=self.parse_question)

    def parse_question(self, response):
        yield {
            'title': response.css('h1 a::text').extract()[0],
            'votes': response.css('.question .vote-count-post::text').extract()[0],
            'body': response.css('.question .post-text').extract()[0],
            'tags': response.css('.question .post-tag::text').extract(),
            'link': response.url,
        }

Answer 1

您不必担心，In [1]: response.urljoin("http://stackoverflow.com/questions/426258/checking-a-checkbox-with-jquery") Out[1]: 'http://stackoverflow.com/questions/426258/checking-a-checkbox-with-jquery' In [2]: response.urljoin("/questions/426258/checking-a-checkbox-with-jquery") Out[2]: 'http://stackoverflow.com/questions/426258/checking-a-checkbox-with-jquery'正确处理这两种情况：

Scrapy Spider：相对链接和绝对链接

1 个答案: