Question

我是R中的网页抓取新手，最近遇到了引用javascript的网站的问题。我试图从下面的网页抓取数据并且未成功。我相信javascript链接阻止我访问该表。因此，R包＆＃34; XML＆＃34;功能＆＃34; readHTMLTable＆＃34;出现无效。

library(XML)
library(RCurl)
url <- "http://votingrights.news21.com/interactive/movement-voter-id/index.html"
tabs <- getURL(url)
tabs <- htmlParse(url)
tabs <- readHTMLTable(tabs, stringsAsFactors = FALSE)

如何访问javascript链接以获取数据？或者这甚至可能吗？当使用直接链接到数据（下面）和R包＆＃34; rjson＆＃34;我仍然无法读入数据。

library("rjson")
json_file <- "http://votingrights.news21.com/static/interactives/movement/data/fulldata.js"
lines <- readLines(json_file)
json_data <- fromJSON(lines, collapse="")

Answer 1

您引用的文件是包含JSON而不是JSON的javascript文件。在这种情况下，您可以手动清理内容以获取数据：

library("rjson")
json_file <- "http://votingrights.news21.com/static/interactives/movement/data/fulldata.js"
lines <- readLines(json_file)
lines[1] <- sub(".* = (.*)", "\\1", lines[1])
lines[length(lines)] <- sub(";", "", lines[length(lines)])
json_data <- fromJSON(paste(lines, collapse="\n"))
> head(json_data[[1]][[1]])
$state
[1] "Alabama"

$bill
[1] "HB 19"

$category
[1] "Strict photo ID"

$introduced
[1] "Mar 1, 2011"

$house
[1] "Yes"

$senate
[1] "Yes"

如果您想与网页上的javascript数据进行互动，可以使用Selenium：

library(RSelenium)
appURL <- "http://votingrights.news21.com/static/interactives/movement/index.html"
pJS <- phantom()
remDr <- remoteDriver(browserName = "phantom")
remDr$open()
remDr$navigate(appURL)
fullData <- remDr$executeScript("return fullData;")
pJS$stop()
> head(fullData[[1]][[1]])
$state
[1] "Alabama"

$bill
[1] "HB 19"

$category
[1] "Strict photo ID"

$introduced
[1] "Mar 1, 2011"

$house
[1] "Yes"

$senate
[1] "Yes"

使用来自R的JavaScript抓取页面

1 个答案: