Question

我正在尝试从需要JavaScript的页面中提取原始HTML（最终目标是仅提取纯文本）。不幸的是，简单的get请求返回HTML，指定运行JS的浏览器的需要。例如：

> html = open('https://www.medicare.gov/Publications/').read
"<!DOCTYPE HTML>\n<!--[if lt IE 8]><html class=\"no-js oldIE\" lang=\"en-US\"><![endif]-->\n<!--[if IE 8]><html class=\"no-js lt-ie9\" lang=\"en-US\"><![endif]-->\n<!--[if IE 9]><html class=\"no-js ie9\" lang=\"en-US\"><![endif]-->\n<!--[if IE 11]>\n<style>\nbody{\ndisplay:none;\n}\n</style>\n<![endif]-->\n<!--[if (gt IE )|!(IE)]><!-->\n<html lang=\"en-US\" class=\"no-js not-ie\">\n    <!--<![endif]-->\n    <head>\n        <meta charset=\"utf-8\">\n        <!--<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">-->\n        <title>Publications</title>...
<body>\n        <div class=\"wrapper sl-translate\">\n            <div class=\"needCSS hidden\">\n                This application is not fully accessible to users whose browsers do not support or have Cascading Style Sheets (CSS) disabled. For a more optimal experience viewing this application, please enable CSS in your browser and refresh the page.\n            </div>\n            <!--<p class=\"browsehappy\">\n                Your browser is out of date  or not supported. Please visit <a href=\"http://browsehappy.com/\">browse happy</a> to upgrade to a better supported, modern browser.\n            </p>-->\n            <div class=\"js-off-message\">\n                <noscript>\n                The page could not be loaded. This application currently does not support browsers with \"JavaScript\" disabled. Please enable JavaScript and refresh the page to view this application.\n                </noscript>"

我试过欺骗网页以为我正在使用带有JS的浏览器，但是（震惊），这也不起作用：

html = = HTTP.headers(user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36").follow.get(url).to_s

从像https://www.medicare.gov/hospicecompare/这样的JS驱动的页面中提取静态HTML是否有任何技巧？

Answer 1

根据建议我使用无头浏览器的评论以及https://readysteadycode.com/howto-scrape-websites-with-ruby-and-headless-chrome中描述的方法，我能够使用以下内容提取HTML和纯文本：

安装ChromeDriver

$ brew install chromedriver

安装selenium-webdriver gem

$ gem install selenium-webdriver

获取网页

> require 'selenium-webdriver'
> options = Selenium::WebDriver::Chrome::Options.new(args: ['headless'])
> driver = Selenium::WebDriver.for(:chrome, options: options)
> driver.get 'https://www.medicare.gov/hospicecompare/'

提取HTML

> driver.find_element(css: 'html').attribute('innerHTML')

提取纯文字

> driver.find_element(css: 'html').text

<强>参考

https://readysteadycode.com/howto-scrape-websites-with-ruby-and-headless-chrome https://github.com/SeleniumHQ/selenium/wiki/Ruby-Bindings

如何从需要JavaScript的URL中提取HTML？

1 个答案: