我想从https://m.youtube.com刮掉评论
当我尝试抓取https://m.youtube.com时,首先将其重定向到https://www.youtube.com。我已对蜘蛛进行了编程,使其不遵守robot.txt,禁用cookie并尝试了meta = dont_redirect。现在它没有将我重定向到https://www.youtube.com,但是现在我得到响应“忽略响应<303 https://m.youtube.com/view_comment?v=xHkL9PU7o9k&gl=US&hl=en&client=mv-google>:HTTP状态码未处理或不允许使用”我该如何解决。
我的蜘蛛密码如下:
import scrapy
class CommentsSpider(scrapy.Spider):
name = 'comments'
allowed_domains = ['m.youtube.com']
start_urls = [
'https://m.youtube.com/view_comment?
v=xHkL9PU7o9k&gl=US&hl=en&client=mvgoogle'
]
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, meta = {'dont_redirect': True})
def parse(self, response):
x = response.xpath('/html/body/div[4]/div[2]/text()').extract()
y =
response.xpath('/html/body/div[4]/div[3]/div[2]/text()').extract()
yield{'Comments': (x, y)}
'''
输出:
2019-07-18 16:07:23 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2019-07-18 16:07:24 [scrapy.core.engine] DEBUG: Crawled (303) <GET https://m.youtube.com/view_comment?v=xHkL9PU7o9k&gl=US&hl=en&client=mv-google> (referer: None)
2019-07-18 16:07:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <303 https://m.youtube.com/view_comment?v=xHkL9PU7o9k&gl=US&hl=en&client=mv-google>: HTTP status code is not handled or not allowed
2019-07-18 16:07:24 [scrapy.core.engine] INFO: Closing spider (finished)
答案 0 :(得分:1)
我会尝试使用移动浏览器的用户代理字符串,以避免被重定向:
USER_AGENT='Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1'
headers = {'User-Agent': USER_AGENT}
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, headers=self.headers)
答案 1 :(得分:0)
According to Scrapy documentation,您可以使用handle_httpstatus_list
蜘蛛属性。
在您的情况下:
class CommentsSpider(scrapy.Spider):
name = 'comments'
allowed_domains = ['m.youtube.com']
start_urls = [
'https://m.youtube.com/view_commentv=xHkL9PU7o9k&gl=US&hl=en&client=mvgoogle'
]
handle_httpstatus_list = [303]