Question

我的数据框D包含文档标题和文本，如下例所示：

document   content
Doc 1      "This is an example of a document"
Doc 2      "And another one"

我需要使用tokenize包中的quanteda函数来标记每个文档，然后返回原始文档标题列出的标记，如下例所示：

document   content
    Doc 1      "This"
    Doc 1      "This is"
    Doc 1      "This is an"
    Doc 1      "This is an example"

这是我目前从文档列表中获取包含令牌的数据框的过程：

require(textreadr)
D<-textreadr::read_dir("myDir")
D<-paste(D$content,collapse=" ")
strlist<-paste0(c(":","\\)",":","'",";","!","+","&","<",">","\\(","\\[","\\]","-","#",","),collapse = "|")
D<-gsub(strlist, "", D)
library(quanteda)
require(quanteda)
t<-tokenize(D, what = c("word","sentence", "character","fastestword", "fasterword"), 
            remove_numbers = FALSE, remove_punct = FALSE,
            remove_symbols = FALSE, remove_separators = TRUE,
            remove_twitter = FALSE, remove_hyphens = FALSE, remove_url = FALSE,
            ngrams = 1:10, concatenator = " ", hash = TRUE,
            verbose = quanteda_options("verbose"))
t<-unlist(t, use.names=FALSE)
t1<-data.frame(t)

但是，我无法找到一种简单的方法来在标记化过程之后保留文档名称并相应地列出标记。任何人都可以帮忙吗？

Answer 1

R的列表对象可以采用如下字符串索引：

my_list = list()

document_title = 'asdf.txt'
my_data = tokenize( etc... )
my_list[[document_title]] = my_data

使用现有代码，但将最终数据框分配到以下列表：

my_list[[document_title]] = data.frame(t)

Answer 2

使用函数了解它的底部。以下是感兴趣的人的代码：

myFunction <- function(x){

b <- x[2]
b<-paste(b,collapse=" ")

require(quanteda)
value <- tokenize(b, what = c("word","sentence", "character","fastestword", "fasterword"), 
                            remove_numbers = FALSE, remove_punct = FALSE,
                            remove_symbols = FALSE, remove_separators = TRUE,
                            remove_twitter = FALSE, remove_hyphens = FALSE, remove_url = FALSE,
                            ngrams = 1:10, concatenator = " ", hash = TRUE,
                            verbose = quanteda_options("verbose"))

value<-unlist(value, use.names=FALSE)

return(value)
        }

D$out <- apply(D, 1, myFunction)

library(tidyr)
D<-unnest(D)

如何通过原始文档标题来标记R中的文档并列出标记？

2 个答案: