我正在使用scrapy 1.6和splash 3.2,我有:
import scrapy
import random
from scrapy_splash import SplashRequest
from scrapy.utils.response import open_in_browser
from scrapy.linkextractors import LinkExtractor
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0'
class MySpider(scrapy.Spider):
start_urls = ["http://yahoo.com"]
name = 'mytest'
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 2.5},headers={'User-Agent': USER_AGENT,'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'})
def parse(self, response):
# response.body is a result of render.html call; it
# contains HTML processed by a browser.
# from scrapy.http.response.html import HtmlResponse
# ht = HtmlResponse('jj')
# ht.body.replace =response
open_in_browser(response)
return None
问题是,当我尝试在浏览器中打开响应时,却在记事本中打开了它。
看着https://splash.readthedocs.io/en/stable/scripting-response-object.html。如何激活response.body,以便可以在浏览器中打开响应(我希望能够使用浏览器开发工具来获取xpath)?
答案 0 :(得分:2)
open_in_browser()
无法将来自Splash的响应检测为HTML响应。这是因为Splash HTML响应对象是Scrapy的TextResponse
的子类,而不是HtmlResponse
(for now)的子类。
您可以暂时以适合您的用例的方式重新实现open_in_browser()
。
答案 1 :(得分:0)
我可以和它一起使用
def parse(self, response):
# response.body is a result of render.html call; it
# contains HTML processed by a browser.
from scrapy.http.response.html import HtmlResponse
ht = HtmlResponse(url=response.url, body=response.body, encoding="utf-8", request=response.request)
open_in_browser(response)
return None