Question

我正在使用rvest包来从网页上抓取信息。我想抓一下网站的前两页http://www.ratemds.com/doctors/?page2=&specialty=family-gp&page=1

以下是我的代码。我面临的问题是刮掉评论部分。哪里有“阅读更多”我想去那个链接并抓取所有评论。

任何人都可以告诉我如何做到这一点？我尝试了follow_link和jump_to，但没有成功。

library(rvest)

hospital <- html("http://www.ratemds.com/doctors/?specialty=family-gp")

nxt_pg<- "&page=3"
nxt_pg1<- "?page2=&"
hospital_test1 <-paste0("http://www.ratemds.com/doctors/",nxt_pg1,"?specialty=family-gp",
                        nxt_pg,sep="") 
hospital_test1 <- html(hospital_test1)

web_crawler(x=hospital)

web_crawler <- function(x)

{
  Doctor_nm <- x %>%  
    html_nodes("h2 a") %>%
    html_text()
  Doctor_nm
  # return(Doctor_nm) 

  Comments <- x %>%
    html_nodes(".rating-comment") %>%
    html_text()
  Comments
  # return(Comments)
  return(list(Doctor_nm,Comments))
}

fd<-web_crawler(x=hospital)
df <- as.data.frame(fd)
colnames(df) <- c ("Doctors_nm","Comments")
fd1<-web_crawler(x=hospital_test1)
df1 <- as.data.frame(fd1)
colnames(df1) <- c ("Doctors_nm","Comments")

Final_data <- rbind(df,df1)

R web scraping-关注＆＃34;阅读更多＆＃34;使用rvest链接并获取详细信息

0 个答案: