Question

我试图废弃一个网站，该网站有一个表格并且可以从JS生成我想要的信息（我猜）。

这是网站：https://www.distancecalculator.net/，它会计算城市之间的距离。

例如，我想计算这两个城市之间的距离：

Craíbas - AL，Brasil
Maceió - AL，Brasil

似乎即使我使用POST来填充表单，我的刮刀仍在收集可用数据，然后单击“计算”按钮。我做错了什么？

这是我的代码：

library(httr)
library(rvest)

url <- "https://www.distancecalculator.net/"

fd <- list(
  submit = "Calculate Distance",
  "originCity"  = "Craíbas - AL, Brasil",
  "destinationCity" = "Maceió - AL, Brasil"
)

resp<-POST(url, body=fd, encode="form")
conte <- content(resp)
conte

tex <- conte %>% html_nodes(xpath = '//span[@id="driving-distance-km"]/text()') %>% html_text()
tex

Answer 1

我同意RSeleneium对此最好的评论。以下是使用RSelenium的理想结果。

library(RSelenium)

url <- "https://www.distancecalculator.net/"

#Start Selenium
rD <- rsDriver(port = 4444L, browser = "chrome")
remDr <- rD$client 
remDr$navigate(url)

#Type in the information
originCity <- remDr$findElement(using = "css", "#originCity")
originCity$sendKeysToElement(list("Craíbas - AL, Brasil"))
#Click the first item
clickFirst <- remDr$findElements(using = "css", ".pac-item")
clickFirst2 <- unlist(lapply(clickFirst, function(x){
  x$getElementText()
}))
clickFirst2
click <- clickFirst[[which(clickFirst2 == clickFirst2[1])]]
click$clickElement()

destinationCity <- remDr$findElement(using = "css", "#destinationCity")
destinationCity$sendKeysToElement(list("Maceió - AL, Brasil"))
#Click the first item
clickFirst <- remDr$findElements(using = "css", ".pac-item")
clickFirst2 <- unlist(lapply(clickFirst, function(x){
  x$getElementText()
}))
clickFirst2
click <- clickFirst[[which(clickFirst2 == clickFirst2[1])]]
click$clickElement()

#No longer Necessary 
calculate <- remDr$findElements(using = "xpath", '//*[contains(concat( " ", 
@class, " " ), concat( " ", "button", " " ))]')
calculate2 <- unlist(lapply(calculate, function(x){
   x$getElementText()
 }))
calculate2
click <- calculate[[which(calculate2 == calculate2[1])]]
click$clickElement()

#Scrape the result
dist <- remDr$findElements(using = "css", "#driving-distance-km")
dist <- unlist(lapply(dist, function(x){
  x$getElementText()
}))
dist
remDr$close()

指向RSelenium包信息的链接：https://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-basics.html

使用R刮取带有表单和JS的网站

1 个答案: