我有很多PDF文档，我已将其读入带有库tm的语料库中。如何将语料库分解成句子？
可以通过在readLines [*]包中使用sentSplit后跟qdap来阅读该文件。该功能需要数据帧。它还需要放弃语料库并单独阅读所有文件。
如何在sentSplit的语料库中传递函数qdap {tm}？或者有更好的方法吗？
注意：库sentDetect中有一个函数openNLP，现在是Maxent_Sent_Token_Annotator - 同样的问题适用：如何将其与语料库[tm]？
答案 0 :(得分：15)
我不知道如何重塑一个语料库，但这将是一个非常棒的功能。
我想我的方法是这样的：
使用这些包
# Load Packages
require(tm)
require(NLP)
require(openNLP)
我将我的文本设置为句子功能如下：
convert_text_to_sentences <- function(text, lang = "en") {
# Function to compute sentence annotations using the Apache OpenNLP Maxent sentence detector employing the default model for language 'en'.
sentence_token_annotator <- Maxent_Sent_Token_Annotator(language = lang)
# Convert text to class String from package NLP
text <- as.String(text)
# Sentence boundaries in text
sentence.boundaries <- annotate(text, sentence_token_annotator)
# Extract sentences
sentences <- text[sentence.boundaries]
# return sentences
return(sentences)
}
我修改了一个重塑语料库功能（注意：你将失去这里的元属性，除非你以某种方式修改这个功能并适当地复制它们）
reshape_corpus <- function(current.corpus, FUN, ...) {
# Extract the text from each document in the corpus and put into a list
text <- lapply(current.corpus, Content)
# Basically convert the text
docs <- lapply(text, FUN, ...)
docs <- as.vector(unlist(docs))
# Create a new corpus structure and return it
new.corpus <- Corpus(VectorSource(docs))
return(new.corpus)
}
其工作原理如下：
## create a corpus
dat <- data.frame(doc1 = "Doctor Who is a British science fiction television programme produced by the BBC. The programme depicts the adventures of a Time Lord—a time travelling, humanoid alien known as the Doctor. He explores the universe in his TARDIS (acronym: Time and Relative Dimension in Space), a sentient time-travelling space ship. Its exterior appears as a blue British police box, a common sight in Britain in 1963, when the series first aired. Along with a succession of companions, the Doctor faces a variety of foes while working to save civilisations, help ordinary people, and right wrongs.",
doc2 = "The show has received recognition from critics and the public as one of the finest British television programmes, winning the 2006 British Academy Television Award for Best Drama Series and five consecutive (2005–10) awards at the National Television Awards during Russell T Davies's tenure as Executive Producer.[3][4] In 2011, Matt Smith became the first Doctor to be nominated for a BAFTA Television Award for Best Actor. In 2013, the Peabody Awards honoured Doctor Who with an Institutional Peabody \"for evolving with technology and the times like nothing else in the known television universe.\"[5]",
doc3 = "The programme is listed in Guinness World Records as the longest-running science fiction television show in the world[6] and as the \"most successful\" science fiction series of all time—based on its over-all broadcast ratings, DVD and book sales, and iTunes traffic.[7] During its original run, it was recognised for its imaginative stories, creative low-budget special effects, and pioneering use of electronic music (originally produced by the BBC Radiophonic Workshop).",
stringsAsFactors = FALSE)
current.corpus <- Corpus(VectorSource(dat))
# A corpus with 3 text documents
## reshape the corpus into sentences (modify this function if you want to keep meta data)
reshape_corpus(current.corpus, convert_text_to_sentences)
# A corpus with 10 text documents
我的sessionInfo输出
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-w64-mingw32/x64 (64-bit)
locale:
[1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] NLP_0.1-0 openNLP_0.2-1 tm_0.5-9.1
loaded via a namespace (and not attached):
[1] openNLPdata_1.5.3-1 parallel_3.0.1 rJava_0.9-4 slam_0.1-29 tools_3.0.1
答案 1 :(得分：5)
openNLP有一些重大变化。坏消息是它看起来与过去非常不同。好消息是它更灵活，你以前享受的功能仍然存在，你只需要找到它。
这将为您提供您所追求的目标：
?Maxent_Sent_Token_Annotator
只需完成示例，您就会看到您正在寻找的功能。
答案 2 :(得分：1)
只需将您的语料库转换为数据框并使用正则表达式来检测句子。
这是一个使用正则表达式来检测段落中的句子并返回每个句子的函数。
chunk_into_sentences <- function(text) {
break_points <- c(1, as.numeric(gregexpr('[[:alnum:] ][.!?]', text)[[1]]) + 1)
sentences <- NULL
for(i in 1:length(break_points)) {
res <- substr(text, break_points[i], break_points[i+1])
if(i>1) { sentences[i] <- sub('. ', '', res) } else { sentences[i] <- res }
}
sentences <- sentences[sentences=!is.na(sentences)]
return(sentences)
}
...在tm包中使用语料库中的一个段落。
text <- paste('Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.')
mycorpus <- VCorpus(VectorSource(text))
corpus_frame <- data.frame(text=unlist(sapply(mycorpus, `[`, "content")), stringsAsFactors=F)
使用如下：
chunk_into_sentences(corpus_frame)
这给了我们：
[1] "Lorem Ipsum is simply dummy text of the printing and typesetting industry."
[2] "Lorem Ipsum has been the industry standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book."
[3] "It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged."
[4] "It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum."
现在拥有更大的语料库
text1 <- "Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum."
text2 <- "It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using 'Content here, content here', making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for 'lorem ipsum' will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like)."
text3 <- "There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don't look even slightly believable. If you are going to use a passage of Lorem Ipsum, you need to be sure there isn't anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc."
text_list <- list(text1, text2, text3)
my_big_corpus <- VCorpus(VectorSource(text_list))
使用如下：
lapply(my_big_corpus, chunk_into_sentences)
这给了我们：
$`1`
[1] "Lorem Ipsum is simply dummy text of the printing and typesetting industry."
[2] "Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book."
[3] "It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged."
[4] "It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum."
$`2`
[1] "It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout."
[2] "The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using 'Content here, content here', making it look like readable English."
[3] "Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for 'lorem ipsum' will uncover many web sites still in their infancy."
$`3`
[1] "There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don't look even slightly believable."
[2] "If you are going to use a passage of Lorem Ipsum, you need to be sure there isn't anything embarrassing hidden in the middle of text."
[3] "All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet."
[4] "It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable."
[5] "The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc."
答案 3 :(得分：0)
使用qdap version 1.1.0，您可以使用以下内容完成此操作（我使用了@Tony Breyal的current.corpus数据集）：
library(qdap)
with(sentSplit(tm_corpus2df(current.corpus), "text"), df2tm_corpus(tot, text))
你也可以这样做：
tm_map(current.corpus, sent_detect)
## inspect(tm_map(current.corpus, sent_detect))
## A corpus with 3 text documents
##
## The metadata consists of 2 tag-value pairs and a data frame
## Available tags are:
## create_date creator
## Available variables in the data frame are:
## MetaID
##
## $doc1
## [1] Doctor Who is a British science fiction television programme produced by the BBC.
## [2] The programme depicts the adventures of a Time Lord—a time travelling, humanoid alien known as the Doctor.
## [3] He explores the universe in his TARDIS, a sentient time-travelling space ship.
## [4] Its exterior appears as a blue British police box, a common sight in Britain in 1963, when the series first aired.
## [5] Along with a succession of companions, the Doctor faces a variety of foes while working to save civilisations, help ordinary people, and right wrongs.
##
## $doc2
## [1] The show has received recognition from critics and the public as one of the finest British television programmes, winning the 2006 British Academy Television Award for Best Drama Series and five consecutive awards at the National Television Awards during Russell T Davies's tenure as Executive Producer.
## [2] In 2011, Matt Smith became the first Doctor to be nominated for a BAFTA Television Award for Best Actor.
## [3] In 2013, the Peabody Awards honoured Doctor Who with an Institutional Peabody for evolving with technology and the times like nothing else in the known television universe.
##
## $doc3
## [1] The programme is listed in Guinness World Records as the longest-running science fiction television show in the world and as the most successful science fiction series of all time—based on its over-all broadcast ratings, DVD and book sales, and iTunes traffic.
## [2] During its original run, it was recognised for its imaginative stor
答案 4 :(得分：0)
错误意味着与ggplot2包连接，annotate函数给出了这个错误，分离ggplot2包然后再试一次。希望它应该工作。
答案 5 :(得分：0)
我使用tokenizers包实现了以下代码来解决同样的问题。
# Iterate a list or vector of strings and split into sentences where there are
# periods or question marks
sentences = purrr::map(.x = textList, function(x) {
return(tokenizers::tokenize_sentences(x))
})
# The code above will return a list of character vectors so unlist
# to give you a character vector of all the sentences
sentences = unlist(sentences)
# Create a corpus from the sentences
corpus = VCorpus(VectorSource(sentences))
答案 6 :(得分：0)
这是一个基于this Python solution的功能，它可以提供一些灵活性，因为可以将前缀，后缀等的列表修改为您的特定文本。它绝对不是完美的，但是在正确的文本中可能很有用。
// Can editable every state with 'changeState' action.
appState.changeState("isMaxBody", true);
appState.changeState("isMobile", true);
appState.changeState("maxBodySize", 960);