Question

我试图从http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D抓取各个提供商的网址。

我查看了网页来源并找出了感兴趣的网址。例如，我想从以下节点中抓取“http://www.childrenshospital.org/doctors/mirna-aeschlimann”

<a data-layer-event="searchClick" data-bind="attr: {href: model.Url}" href="http://www.childrenshospital.org/doctors/mirna-aeschlimann"><!--ko text: model.FirstName-->Mirna<!--/ko--><!--ko text: ' ' + model.LastName--> Aeschlimann<!--/ko--><!--ko if: model.Suffix-->, <!--ko text: model.Suffix-->MD<!--/ko--><!--/ko--></a>

我尝试了以下代码。但是，由于某些原因，它没有返回上面的节点。

base_html <- "http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D"
doc <- htmlTreeParse(base_html, useInternal = TRUE)

非常感谢任何帮助。如果我应该提供进一步的信息，请告诉我。

Answer 1

您是否尝试定位网站为获取数据而发出的XHR请求？

library(httr)
library(purrr)
library(xml2)
library(dplyr)
library(jsonlite)

map_df(1:17, function(i) {

  POST("http://www.childrenshospital.org/searchdirectory.ajax",
       body = list(search_query = "",
                   search_specialties = "",
                   search_languages = "",
                   search_gender = "", search_departments = "",
                   search_programs = "", search_userlocation = "",
                   search_radius = "10", search_pcp = "true",
                   search_specialist = "false",
                   search_type = "directorynurse|directoryphysician",
                   search_letter = "", search_querygroup = "dirnametext",
                   search_page = "10"),
       encode = "form") -> res

  content(res, as="text") %>%
    fromJSON() %>%
    .$Records %>%
    mutate(Address=xml2::xml_text(xml2::read_html(paste0("<x>", Address, "</x>")))) %>%
    tbl_df()

}) -> tmp_df

glimpse(tmp_df)
## Observations: 408
## Variables: 21
## $ ID             <chr> "{E8ECAF3B-B49C-4CD8-AB16-6CE63F0379C0}", "{1E1...
## $ FirstName      <chr> "Jonathan", "Barbara", "Mark", "Maura", "Sarah"...
## $ LastName       <chr> "Schwab", "Seagle", "Shapira", "Shea", "Sheldon...
## $ Image          <chr> "/~/media/directory/physicians/schwab_jonathan....
## $ Suffix         <chr> "MD", "MD", "MD", "MD", "MD", "MD", "MD", "MD",...
## $ Url            <chr> "http://www.childrenshospital.org/doctors/jonat...
## $ Gender         <chr> "male", "female", "male", "female", "female", "...
## $ Latitude       <chr> "42.3344382", "42.326435", "41.559642", "42.423...
## $ Longitude      <chr> "-72.6618324", "-71.149499", "-70.939315", "-71...
## $ Address        <chr> "{\"practice_name\":\"Northampton Area Pediatri...
## $ Distance       <chr> "", "", "", "", "", "", "", "", "", "", "", "",...
## $ OtherLocations <chr> "", "", "", "", "", "", "", "", "", "", "Westwo...
## $ AcademicTitle  <chr> "", "", "", "", "", "", "", "", "", "", "", "",...
## $ HospitalTitle  <chr> "Pediatrician", "Pediatrician", "Pediatrician",...
## $ Specialties    <chr> "Pediatrics", "General Pediatrics, Pediatrics, ...
## $ Departments    <chr> "", "General Pediatrics", "General Pediatrics",...
## $ Languages      <chr> "", "English", "", "English", "English", "", "E...
## $ PPOCLink       <chr> "http://www.childrenshospital.org/patient-resou...
## $ Gallery        <chr> "", "", "", "", "", "", "", "", "", "", "", "",...
## $ Phone          <chr> "(413) 584-8700", "(617) 731-0200", "(508) 996-...
## $ Fax            <chr> "(413) 584-1714", "(617) 731-0289", "(508) 992-...

使用R中的JavaScript从网页抓取链接

1 个答案: