Question

我试图抓取一个论坛，最终发布帖子中包含已发布链接的帖子。现在我只是试图抓住帖子的用户名。但我认为网址存在问题，而不是静态的。

spider.py

from scrapy.spiders import CrawlSpider
from scrapy.selector import Selector
from scrapy.item import Item, Field


class TextPostItem(Item):
    title = Field()
    url = Field()
    submitted = Field()


class RedditCrawler(CrawlSpider):
    name = 'post-spider'
    allowed_domains = ['flashback.org']
    start_urls = ['https://www.flashback.org/t2637903']


    def parse(self, response):
        s = Selector(response)
        next_link = s.xpath('//a[@class="smallfont2"]//@href').extract()[0]
        if len(next_link):
            yield self.make_requests_from_url(next_link)
        posts =   Selector(response).xpath('//div[@id="posts"]/div[@class="alignc.p4.post"]')
        for post in posts:
            i = TextPostItem()
            i['title'] = post.xpath('tbody/tr[1]/td/span/text()').extract() [0]
            #i['url'] = post.xpath('div[2]/ul/li[1]/a/@href').extract()[0]
            yield i

提供以下错误：

raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: /t2637903p2

有什么想法吗？

Answer 1

您需要将response.url与您使用urljoin()提取的相对网址“加入”：

from urlparse import urljoin

urljoin(response.url, next_link)

另请注意，无需实例化Selector对象 - 您可以直接使用response.xpath()快捷方式：

def parse(self, response):
    next_link = response.xpath('//a[@class="smallfont2"]//@href').extract()[0]
    # ...

与scrapy绝对的相对路径

1 个答案: