Question

我正在尝试提取纽约时代电影评论的主体，以便对它们进行一些语义分析。不幸的是，我的HTML + R + XML包技能还不足以完成工作。我可以使用NYT电影API的XML输出来获取电影详细信息，但我无法弄清楚如何使用文章API或直接网页抓取，以便进入评论的主体。

工作代码到电影细节：

library(RCurl)
nyt.x.url<-'http://api.nytimes.com/svc/movies/v2/reviews/search.xml?query=The+Hangover&api-key=YOUR-OWN-FREE-API-KEY-GOES-HERE'
nyt.x.out<-getURLContent(nyt.x.url,curl=getCurlHandle())
library(XML)
a <- xmlTreeParse(nyt.x.url)
r <- xmlRoot(a)
# need to put the separate list items together into a mtrix, before they can be turned to a dataframe
nyt.df <- as.data.frame(stringsAsFactors=FALSE,
                    matrix(c(as.character(r[[4]][[1]][[1]][[1]])[6],  # display name
                             as.character(r[[4]][[1]][[3]][[1]])[6],  # rating - agrees with rotten tomatoes, but not imdb
                             as.character(r[[4]][[1]][[4]][[1]])[6],  # is it a critics pick
                             as.character(r[[4]][[1]][[5]][[1]])[6],  # is it a thousand best
                             as.character(r[[4]][[1]][[11]][[1]])[6],  # opening date
                             as.character(r[[4]][[1]][[15]][[1]][[1]])[6]),  # this is really the URL....
                           nrow=1,
                           ncol=6))

# now apply the right names
colnames(nyt.df) <- c("Title","MPAA-Rating", "Critics.Pick", "Thousand.Best", "Release.Date", "Article.URL")

然后我会使用这个电影详情的数据框来抓住评论网页并尝试抓住评论文字：

nyt.review.out<-getURLContent(as.character(nyt.df[6]),curl=getCurlHandle())
a2 <- htmlTreeParse(nyt.review.url)

但我无法弄清楚如何查看评论的全文。当我尝试将json API用于文章时遇到同样的问题（对api的url调用如下）

nyt.review.url＆lt; - 'http://api.nytimes.com/svc/search/v1/article?format=json&query=review+the+Hangover&begin_date=20090605&end_date=20090606& API-键= YOUR-OTHER-FREE-API-KEY-GOES-HERE'

非常感谢任何帮助，但您需要注册自己的API密钥（我已从代码中删除了我的）

Answer 1

我认为这可以做你想要的。可能有一种方法可以直接从API中执行您想要的操作，但我没有对此进行调查。

# load package
library(XML)

# grabs text from new york times movie page. 
grab_nyt_text <- function(u) {
  doc <- htmlParse(u)
  txt <- xpathSApply(doc, '//div[@class="articleBody"]//p', xmlValue)
  txt <- paste(txt, collapse = "\n")
  free(doc)
  return(txt)
}


###--- Main ---###

# Step 1: api URL
nyt.x.url <- 'http://api.nytimes.com/svc/movies/v2/reviews/search.xml?query=The+Hangover&api-key=YOUR-OWN-FREE-API-KEY-GOES-HERE'

# Step 2: Parse XML of webpage pointed to by URL
doc <- xmlParse(nyt.x.url)

# Step 3: Parse XML and extract some values using XPath expressions
df <- data.frame(display.title = xpathSApply(doc, "//results//display_title", xmlValue), 
                 critics.pick = xpathSApply(doc, "//results//critics_pick", xmlValue),
                 thousand.best = xpathSApply(doc, "//results//thousand_best", xmlValue),
                 opening.date = xpathSApply(doc, "//results//opening_date", xmlValue),
                 url = xpathSApply(doc, "//results//link[@type='article']/url", xmlValue),
                 stringsAsFactors=FALSE)

df
#         display.title critics.pick thousand.best opening.date                                                                                           url
#1         The Hangover            0             0   2009-06-05                                       http://movies.nytimes.com/2009/06/05/movies/05hang.html
#2 The Hangover Part II            0             0   2011-05-26 http://movies.nytimes.com/2011/05/26/movies/the-hangover-part-ii-3-men-and-a-monkey-baby.html

# Step 4: clean up - remove doc from memory
free(doc)

# Step 5: crawl article links and grab text
df$text <- sapply(df$url, grab_nyt_text)

# Step 6: inspect txt
cat(df$text[1])

HTH

Tony Breyal

P.S。还有一个R包http://www.omegahat.org/RNYTimes，但网站现在已经关闭，所以我不知道它有什么能力。

无法使用R XML包从已删除的HTML页面中提取文本

1 个答案: