Scrapy 404错误：抓取网页时未处理或不允许HTTP状态代码

Question

我正在尝试使用Scrapy抓取subreddit，但是每次运行Spider时，我都会不断收到404错误。

2020-01-07 12:21:46 [scrapy.spidermiddlewares.httperror]信息：忽略响应<404 https://www.reddit.com/r/gameofthrones//>：未处理或不允许HTTP状态代码

我当前正在使用的代码：

import scrapy 
class RedditbotSpider(scrapy.Spider):
    name = 'redditbot'
    allowed_domains = ['www.reddit.com/r/gameofthrones/']
    start_urls = ['http://www.reddit.com/r/gameofthrones//']

    def parse(self, response):
        #Extracting the content using css selectors
        titles = response.css('.title.may-blank::text').extract()
        votes = response.css('.score.unvoted::text').extract()
        times = response.css('time::attr(title)').extract()
        comments = response.css('.comments::text').extract()

        #Give the extracted content row wise
        for item in zip(titles,votes,times,comments):
            #create a dictionary to store the scraped info
            scraped_info = {
                'title' : item[0],
                'vote' : item[1],
                'created_at' : item[2],
                'comments' : item[3],
            }

            #yield or give the scraped info to scrapy
            yield scraped_info

我在settings.py文件中更改了USER_AGENT后尝试重新运行，但是我遇到了同样的问题。

Answer 1

检查您的URL ... http://www.reddit.com/r/gameofthrones//（<-双斜杠），因为您的起始URL不存在，并引发404错误。