Question

我对使用R进行搜索有些新意，但我收到的错误信息是我无法理解的。我的代码：

 url <- "https://en.wikipedia.org/wiki/California_State_Legislature,_2017%E2%80%9318_session"

leg <- read_html(url)

testdata <- leg %>% 
  html_nodes('table') %>% 
  .[6] %>% 
  html_table()

我得到了回复：

out [j + k，]：下标超出范围
时出错

当我用html_text换出html_table时，我没有收到错误。知道我做错了什么吗？

谢谢！

Answer 1

为什么不更好地定位表？

library(rvest)

wp_url <- "https://en.wikipedia.org/wiki/California_State_Legislature,_2017%E2%80%9318_session"

leg <- read_html(wp_url)

html_node(leg, xpath=".//table[contains(., 'District')]") %>%
  html_table()
##            Position                   Position                   Name                  Party               District
## 1                          Lieutenant Governor           Gavin Newsom             Democratic                       
## 2                        President pro tempore          Kevin de León             Democratic       24th–Los Angeles
## 3                              Majority leader           Bill Monning             Democratic            17th–Carmel
## 4                                Majority whip          Nancy Skinner             Democratic           9th–Berkeley
## 5                        Majority caucus chair           Connie Leyva             Democratic             20th–Chino
## 6                   Majority caucus vice chair           Mike McGuire             Democratic         2nd–Healdsburg
## 7                              Minority leader         Patricia Bates             Republican     36th–Laguna Niguel
## 8                        Minority caucus chair            Jim Nielsen             Republican             4th–Gerber
## 9                                Minority whip             Ted Gaines             Republican    1st–El Dorado Hills
## 10        Secretary                  Secretary         Daniel Alvarez         Daniel Alvarez         Daniel Alvarez
## 11 Sergeant-at-Arms           Sergeant-at-Arms         Debbie Manning         Debbie Manning         Debbie Manning
## 12         Chaplain                   Chaplain Sister Michelle Gorman Sister Michelle Gorman Sister Michelle Gorman

哎呀！错误的桌子。仅使用这样的数字索引仍然是不明智的。我们仍然可以更好地定位您想要的表格：

library(rvest)
library(purrr)

wp_url <- "https://en.wikipedia.org/wiki/California_State_Legislature,_2017%E2%80%9318_session"

leg <- read_html(wp_url)

target_table <- html_node(leg, xpath=".//span[@id='Members']/../following-sibling::table")

但是，rvest::html_table()导致错误，你绝对应该在GH页面上提交错误报告。

另一个答案中使用的htmltab pkg看起来很方便（并且可以接受这个答案，因为它更短并且有效）。

我们将采用传统方式，但需要一个辅助函数来制作更好的列名：

mcga <- function(x) {
  x <- tolower(x)
  x <- gsub("[[:punct:][:space:]]+", "_", x)
  x <- gsub("_+", "_", x)
  x <- gsub("(^_|_$)", "", x)
  make.unique(x, sep = "_")
}

现在，我们提取标题行和数据行：

header_row <- html_node(target_table, xpath=".//tr[th]")
data_rows <- html_nodes(target_table, xpath=".//tr[td]")

我们偷看标题行，看到那里有一个邪恶的colspan。我们稍后会利用这些知识。

html_children(header_row)
## {xml_nodeset (6)}
## [1] <th scope="col" width="30" colspan="2">District</th>
## [2] <th scope="col" width="170">Name</th>
## [3] <th scope="col" width="70">Party</th>
## [4] <th scope="col" width="130">Residence</th>
## [5] <th scope="col" width="60">Term-limited?</th>
## [6] <th scope="col" width="370">Notes</th>

获取列名称，并使它们整洁：

html_children(header_row) %>%
  html_text() %>%
  tolower() %>%
  mcga() -> col_names

现在，遍历行，拉出值，删除额外的第一个值并将整个事物转换为数据框：

map_df(data_rows, ~{
  kid_txt <- html_children(.x) %>% html_text() 
  as.list(setNames(kid_txt[-1], col_names))
})
## # A tibble: 40 x 6
##    district              name      party       residence term_limited notes
##       <chr>             <chr>      <chr>           <chr>        <chr> <chr>
##  1        1        Ted Gaines Republican El Dorado Hills                   
##  2        2      Mike McGuire Democratic      Healdsburg                   
##  3        3         Bill Dodd Democratic            Napa                   
##  4        4       Jim Nielsen Republican          Gerber                   
##  5        5 Cathleen Galgiani Democratic        Stockton                   
##  6        6       Richard Pan Democratic      Sacramento                   
##  7        7      Steve Glazer Democratic          Orinda                   
##  8        8     Tom Berryhill Republican     Twain Harte          Yes      
##  9        9     Nancy Skinner Democratic        Berkeley                   
## 10       10    Bob Wieckowski Democratic         Fremont                   
## # ... with 30 more rows

Answer 2

希望这有帮助！

library(htmltab)
library(dplyr)
library(tidyr)

url <- "https://en.wikipedia.org/wiki/California_State_Legislature,_2017%E2%80%9318_session"
url %>%
  htmltab(6, rm_nodata_cols = F) %>%
  .[,-1] %>%
  replace_na(list(Notes = "", "Term-limited?" = "")) %>%
  `rownames<-` (seq_len(nrow(.)))

输出是：

  District              Name      Party       Residence Term-limited? Notes
1        1        Ted Gaines Republican El Dorado Hills                    
2        2      Mike McGuire Democratic      Healdsburg                    
3        3         Bill Dodd Democratic            Napa                    
4        4       Jim Nielsen Republican          Gerber                    
5        5 Cathleen Galgiani Democratic        Stockton                    
6        6       Richard Pan Democratic      Sacramento                    
...

Rvest html_table错误 - out [j + k，]：下标超出界限时出错

2 个答案: