Question

我有大量的取水许可证可在线获取，我想从中提取一些数据。例如

url <- "https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1"

我一点都不懂html，但是一直在Google和一位朋友的帮助下插手。我可以使用xpath或css选择器到达某些节点，而不会出现任何问题，例如，获得标题：

library(rvest)
url %>% 
read_html() %>% 
html_nodes(xpath = '//*[@id="main"]/div/h1') %>%
html_text()
[1] "Details for CRC000002.1"

或使用CSS选择器：

url %>% 
read_html() %>% 
html_nodes(css = "#main") %>% 
html_nodes(css = "div") %>% 
html_nodes(css = "h1") %>% 
html_text()
[1] "Details for CRC000002.1"

到目前为止，还不错，但是我真正想要的信息被深层掩盖了，我似乎无法理解。例如，客户名称字段（在本例中为“ Killermont Station Limited”）具有以下xpath：

clientxpath <- '//*[@id="main"]/div/div[1]/div/table/tbody/tr[1]/td[2]'
url %>% 
read_html() %>% 
html_nodes(xpath = clientxpath) %>%
html_text()
character(0)

css选择器变得相当复杂，但是我得到了相同的结果。 html_nodes（）的帮助文件说：

# XPath selectors ---------------------------------------------
# chaining with XPath is a little trickier - you may need to vary
# the prefix you're using - // always selects from the root noot
# regardless of where you currently are in the doc

但是它并没有为我提供任何在xpath中使用替代前缀的线索（如果我知道html的话，可能会很明显）。

我的朋友指出，某些文档位于javascript（ajax）中，这也可能是问题的一部分。就是说，我要介绍的内容在html中显示，但它位于名为“ div.ajax-block”的节点内。

css selectors: #main > div > div.ajax-block > div > table > tbody > tr:nth-child(1) > td:nth-child(4)

有人可以帮忙吗？谢谢！

Answer 1

非常令人不安的是，大多数（即使不是全部）SO R贡献者默认在刮除“答案”时默认“使用重量级第三方依赖关系”。 99％的时间您不需要硒。您只需要锻炼灰色小细胞即可。

首先，出现的 big 线索是页面异步加载内容。第二个是在您的代码段中，其中div实际上具有选择器名称的一部分，其中包含ajax。正在进行XHR请求的告示牌。

如果在浏览器中打开“开发人员工具”并重新加载页面，然后转到“网络”，然后转到“ XHR”标签，您将看到：

页面上的大多数“真实”数据都是动态加载的。我们可以编写模仿浏览器调用的httr调用。

但是…

我们首先需要对主页进行一次GET调用，以灌注一些cookie，这些cookie将留给我们使用，然后找到每个生成的会话令牌，用于防止对该站点的滥用。它是使用JavaScript定义的，因此我们将使用V8包对其进行评估。我们可以使用正则表达式来查找字符串。做你喜欢的事。

library(httr)
library(rvest)
library(dplyr)
library(V8)

ctx <- v8() # we need this to eval some javascript

# Prime Cookies -----------------------------------------------------------

res <- httr::GET("https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1")

httr::cookies(res)
##          domain flag path secure          expiration                 name
## 1 .ecan.govt.nz TRUE    /  FALSE 2019-11-24 11:46:13   visid_incap_927063
## 2 .ecan.govt.nz TRUE    /  FALSE                <NA> incap_ses_148_927063
##                                                              value
## 1 +p8XAM6uReGmEnVIdnaxoxWL+VsAAAAAQUIPAAAAAABjdOjQDbXt7PG3tpBpELha
## 2         nXJSYz8zbCRj8tGhzNANAhaL+VsAAAAA7JyOH7Gu4qeIb6KKk/iSYQ==
pg <- httr::content(res)

html_node(pg, xpath=".//script[contains(., '_monsido')]") %>%
  html_text() %>%
  ctx$eval()
## [1] "2"
monsido_token <- ctx$get("_monsido")[1,2]

这是searchlist（实际上是空的）：

httr::VERB(
  verb = "POST", url = "https://www.ecan.govt.nz/data/document-library/searchlist",
  httr::add_headers(
    Referer = "https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1",
    `X-Requested-With` = "XMLHttpRequest",
    TE = "Trailers"
  ), httr::set_cookies(
    monsido = monsido_token
  ),
  body = list(
    name = "CRC000002.1",
    pageSize = "999999"
  ),
  encode = "form"
) -> res

httr::content(res)
## NULL ## <<=== this is OK as there is no response

这是“同意概述”部分：

httr::GET(
  url = "https://www.ecan.govt.nz/data/consent-search/consentoverview/CRC000002.1",
  httr::add_headers(
    Referer = "https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1",
    Authority = "www.ecan.govt.nz",
    `X-Requested-With` = "XMLHttpRequest"
  ),
  httr::set_cookies(
    monsido = monsido_token
  )
) -> res

httr::content(res) %>%
  html_table() %>%
  glimpse()
## List of 1
##  $ :'data.frame':    5 obs. of  4 variables:
##   ..$ X1: chr [1:5] "RMA Authorisation Number" "Consent Location" "To" "Commencement Date" ...
##   ..$ X2: chr [1:5] "CRC000002.1" "Manuka Creek, KILLERMONT STATION" "To take water from Manuka Creek at or about map reference NZMS 260 H39:5588-2366 for irrigation of up to 40.8 hectares." "29 Apr 2010" ...
##   ..$ X3: chr [1:5] "Client Name" "State" "To take water from Manuka Creek at or about map reference NZMS 260 H39:5588-2366 for irrigation of up to 40.8 hectares." "29 Apr 2010" ...
##   ..$ X4: chr [1:5] "Killermont Station Limited" "Issued - Active" "To take water from Manuka Creek at or about map reference NZMS 260 H39:5588-2366 for irrigation of up to 40.8 hectares." "29 Apr 2010" ...

以下是“同意条件”：

httr::GET(
  url = "https://www.ecan.govt.nz/data/consent-search/consentconditions/CRC000002.1",
  httr::add_headers(
    Referer = "https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1",
    Authority = "www.ecan.govt.nz",
    `X-Requested-With` = "XMLHttpRequest"
  ),
  httr::set_cookies(
    monsido = monsido_token
  )
) -> res

httr::content(res) %>%
  as.character() %>%
  substring(1, 300) %>%
  cat()
## <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
## <html><body><div class="consentDetails">
##     <ul class="unstyled-list">
## <li>
##           
##             
##             <strong class="pull-left">1</strong> <div class="pad-left1">The rate at which wa

这是“同意相关”：

httr::GET(
  url = "https://www.ecan.govt.nz/data/consent-search/consentrelated/CRC000002.1",
  httr::add_headers(
    Referer = "https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1",
    Authority = "www.ecan.govt.nz",
    `X-Requested-With` = "XMLHttpRequest"
  ),
  httr::set_cookies(
    monsido = monsido_token
  )
) -> res

httr::content(res) %>%
  as.character() %>%
  substring(1, 300) %>%
  cat()
## <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
## <html><body>
## <p>There are no related documents.</p>
## 
## 
## 
## 
##   
##     <div class="summary-table-wrapper">
##       <table class="summary-table left">
## <thead><tr>
## <th>Relationship</th>
##           <th>Recor

这是“工作流程：

httr::GET(
  url = "https://www.ecan.govt.nz/data/consent-search/consentworkflow/CRC000002.1",
  httr::add_headers(
    Referer = "https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1",
    Authority = "www.ecan.govt.nz",
    `X-Requested-With` = "XMLHttpRequest"
  ),
  httr::set_cookies(
    monsido = monsido_token
  )
) -> res

httr::content(res)
## {xml_document}
## <html>
## [1] <body><p>No workflow</p></body>

以下是“允许流量限制”：

httr::GET(
  url = "https://www.ecan.govt.nz/data/consent-search/consentflowrestrictions/CRC000002.1",
  httr::add_headers(
    Referer = "https://www.ecan.govt.nz/data/consent-search/consentdetails/CRC000002.1",
    Authority = "www.ecan.govt.nz",
    `X-Requested-With` = "XMLHttpRequest"
  ),
  httr::set_cookies(
    monsido = monsido_token
  )
) -> res

httr::content(res) %>%
  as.character() %>%
  substring(1, 300) %>%
  cat()
## <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
## <html><body><div class="summary-table-wrapper">
##     <table class="summary-table left">
## <thead>
## <th colspan="2">Low Flow Site</th>
##       <th>Todays Flow <span class="lower">(m3/s)</span>
## </th>

您仍然需要解析HTML，但是现在您只需使用普通的R包就可以完成所有操作。

使用R提取HTML文本-无法访问某些节点

1 个答案: