如何在R中获取简单的HTML表单?

时间:2015-12-27 10:22:45

标签: html r get rcurl rvest

我有一个类似的问题。我正在尝试从美国人口普查地理编码器链接获取地址的坐标(纬度和经度)。我遵循了here提到的方法;但是,我没有得到所需的结果。让我说明我在3次尝试中遵循的步骤:

尝试#1(使用RCurl):

url_geo <- "http://geocoding.geo.census.gov/geocoder/locations/address?form"
td.html <- getForm(url_geo,
submit = "Find",
street  = "3211 Providence Dr",
city = "Anchorage",
state   = "AK",
zip = "99508",
benchmark = "Public_AR_Current",
.opts = curlOptions(ssl.verifypeer = FALSE))

当我看到td.html的输出时,它与您在上面的网页上执行“查看页面来源”时获得的输出相同。实际上,td.html应该包含在上面的网页中提交表单后出现的结果页面的详细信息。

尝试#2(使用httr):

url_geo <- "http://geocoding.geo.census.gov/geocoder/locations/address?form"
fd1 <- list(
submit = "Find",
street  = "3211 Providence Dr",
city = "Anchorage",
state   = "AK",
zip = "99508",
benchmark = "Public_AR_Current"
)
resp1<-GET(url_geo, body=fd1, encode="form")
content(resp1)

resp1的内容与人们期望的非常不同。

尝试#3(使用rvest):

url_geo <- "http://geocoding.geo.census.gov/geocoder/locations/address?form"
s <- html_session(url_geo)
f0 <- html_form(s)

在这里,我收到一个错误:

  

错误:当前页面似乎不是html。

请帮助我理解我做错了什么。如果您需要我的任何澄清,请告诉我。

2 个答案:

答案 0 :(得分:3)

人口普查网站足以让你回到JSON(这是意料之外的,并且通过此次通话获得了很好的奖励):

library(httr)
library(jsonlite)

URL <- "http://geocoding.geo.census.gov/geocoder/locations/address"

res <- GET(URL,
           query=list(street="3211 Providence Dr",
                      city="Anchorage",
                      state="AK",
                      zip="99508",
                      benchmark=4))

dat <- fromJSON(content(res, as="text"))

str(dat$result$addressMatches)
## 'data.frame': 1 obs. of  4 variables:
##  $ matchedAddress   : chr "3211 PROVIDENCE DR, ANCHORAGE, AK, 99508"
##  $ coordinates      :'data.frame':  1 obs. of  2 variables:
##   ..$ x: num -150
##   ..$ y: num 61.2
##  $ tigerLine        :'data.frame':  1 obs. of  2 variables:
##   ..$ tigerLineId: chr "638504877"
##   ..$ side       : chr "L"
##  $ addressComponents:'data.frame':  1 obs. of  12 variables:
##   ..$ fromAddress    : chr "3001"
##   ..$ toAddress      : chr "3399"
##   ..$ preQualifier   : chr ""
##   ..$ preDirection   : chr ""
##   ..$ preType        : chr ""
##   ..$ streetName     : chr "PROVIDENCE"
##   ..$ suffixType     : chr "DR"
##   ..$ suffixDirection: chr ""
##   ..$ suffixQualifier: chr ""
##   ..$ city           : chr "ANCHORAGE"
##   ..$ state          : chr "AK"
##   ..$ zip            : chr "99508"

您可以使用flatten参数fromJSON来处理数据框架中可怕数据结构中的数据框:

dat <- fromJSON(content(res, as="text"), flatten=TRUE)
dplyr::glimpse(dat$result$addressMatches)

## Observations: 1
## Variables: 17
## $ matchedAddress                    (chr) "3211 PROVIDENCE DR, ANCHORAGE, AK, 99508"
## $ coordinates.x                     (dbl) -149.8188
## $ coordinates.y                     (dbl) 61.18985
## $ tigerLine.tigerLineId             (chr) "638504877"
## $ tigerLine.side                    (chr) "L"
## $ addressComponents.fromAddress     (chr) "3001"
## $ addressComponents.toAddress       (chr) "3399"
## $ addressComponents.preQualifier    (chr) ""
## $ addressComponents.preDirection    (chr) ""
## $ addressComponents.preType         (chr) ""
## $ addressComponents.streetName      (chr) "PROVIDENCE"
## $ addressComponents.suffixType      (chr) "DR"
## $ addressComponents.suffixDirection (chr) ""
## $ addressComponents.suffixQualifier (chr) ""
## $ addressComponents.city            (chr) "ANCHORAGE"
## $ addressComponents.state           (chr) "AK"
## $ addressComponents.zip             (chr) "99508"

这将它包装成一个更容易调用的函数:

#' Geocode address using the Census API
#'
#' @param steet Street
#' @param city City
#' @param state State
#' @param zip Zip code
#' @param benchmark "\code{current}" for this most current information,
#'        "\code{2014}" for data from the 2014 U.S. ACS survey,
#'        "\code{2010}" for data from the 2010 U.S. Census. This defaults
#'        to "\code{current}".
#' @result \code{list} of query params and response values. If successful,
#'         the geocoded values will be in \code{var$result$addressMatches}
census_geocode <- function(street, city, state, zip, benchmark="current") {

  URL <- "http://geocoding.geo.census.gov/geocoder/locations/address"

  bench <- c(`current`=4, `2014`=8, `2010`=9)[benchmark]

  res <- GET(URL,
             query=list(street=street, city=city, state=state,
                        zip=zip, benchmark=bench))

  warn_for_status(res)

  fromJSON(content(res, as="text"), flatten=TRUE)

}

census_geocode("3211 Providence Dr", "Anchorage", "AK", "99508")

答案 1 :(得分:0)

构建您的网址并直接提交生成的网址,绕过任何表单!例如,使用您选择的参数,您将获得以下URL:

urlgeo<-"http://geocoding.geo.census.gov/geocoder/locations/address?street=3211+Providence+Dr&city=Anchorage&state=AK&zip=99508&benchmark=4"

然后,您只需通过getURL

检索内容即可
getURL(urlgeo)

将拥有所有必需的信息。要构建网址,只需paste个参数,用+替换任何空格。