Question

我正在处理一个大型的1M doc语料库，并在从中创建文档频率矩阵时应用了几个转换：

library(quanteda)
corpus_dfm <- dfm(tokens(corpus1M), # where corpus1M is already a corpus via quanteda::corpus()
                  remove = stopwords("english"),
                  #what = "word", #experimented if adding this made a difference
                  remove_punct = T,
                  remove_numbers = T,
                  remove_symbols = T,
                  ngrams = 1:2,
                  dictionary = lut_dict,
                  stem = TRUE)

然后查看结果特征：

dimnames(corpus_dfm)$features
[1] "abandon"                                      
[2] "abandoned auto"                               
[3] "abandoned vehicl"
...
[8] "accident hit and run"
...
[60] "assault no weapon aggravated injuri"

为什么这些功能超过1：2 bigrams长？词干似乎是成功的，但令牌似乎是句子而不是单词。

我尝试将代码调整为：dfm(tokens(corpus1M, what = "word")但没有变化。

我试图制作一个可重复的小例子：

library(tidyverse) # just for the pipe here
example_text <- c("the quick brown fox",
                  "I like carrots",
                  "the there that etc cats dogs") %>% corpus

然后，如果我应用与上面相同的dfm：

> dimnames(corpus_dfm)$features
[1] "etc."

这是令人惊讶的，因为几乎所有的单词都被删除了？甚至不像之前的停顿词，所以我更困惑！尽管只是尝试，我现在也无法创建一个可重现的例子。也许我误解了这个功能是如何运作的？

如何在quanteda中创建一个dfm，其中只有1：2个单词标记，并且删除了停用词？

Answer 1

第一个问题：为什么dfm中的功能（名称）这么长？

答案：因为dfm()调用中字典的应用用字典键替换了你的unigrams和bigram特征的匹配，并且（很多）字典中的键由多个单词组成。例如：

lut_dict[70:72]
# Dictionary object with 3 key entries.
# - assault felony:
#     - asf
# - assault misdemeanor:
#     - asm
# - assault no weapon aggravated injury:
#     - anai

第二个问题：在可重现的例子中，为什么几乎所有单词都消失了？

答案：因为字典值与dfm中的功能的唯一匹配是“等”类别。

corpus_dfm2 <- dfm(tokens(example_text), # where corpus1M is already a corpus via quanteda::corpus()
                  remove = stopwords("english"),
                  remove_punct = TRUE,
                  remove_numbers = TRUE,
                  remove_symbols = TRUE,
                  dictionary = lut_dict,
                  ngrams = 1:2,
                  stem = TRUE, verbose = TRUE)
corpus_dfm2
# Document-feature matrix of: 3 documents, 1 feature (66.7% sparse).
# 3 x 1 sparse Matrix of class "dfmSparse"
#        features
# docs    etc.
#   text1    0
#   text2    0
#   text3    1

lut_dict["etc."]
# Dictionary object with 1 key entry.
# - etc.:
#     - etc

如果您不应用字典，那么您会看到：

dfm(tokens(example_text),   # the "tokens" is not necessary here
    remove = stopwords("english"),
    remove_punct = TRUE,
    remove_numbers = TRUE,
    remove_symbols = TRUE,
    ngrams = 1:2,
    stem = TRUE)
# Document-feature matrix of: 3 documents, 18 features (66.7% sparse).
# 3 x 18 sparse Matrix of class "dfmSparse"
#        features
# docs    quick brown fox the_quick quick_brown brown_fox like carrot i_like
#   text1     1     1   1         1           1         1    0      0      0
#   text2     0     0   0         0           0         0    1      1      1
#   text3     0     0   0         0           0         0    0      0      0
#        features
# docs    like_carrot etc cat dog the_there there_that that_etc etc_cat cat_dog
#   text1           0   0   0   0         0          0        0       0       0
#   text2           1   0   0   0         0          0        0       0       0
#   text3           0   1   1   1         1          1        1       1       1

如果您希望保持功能不匹配，请将dictionary替换为thesaurus。下面，您将看到“etc”标记已被替换为大写的“ETC”。

dfm(tokens(example_text), 
    remove = stopwords("english"),
    remove_punct = TRUE,
    remove_numbers = TRUE,
    remove_symbols = TRUE,
    thesaurus = lut_dict,
    ngrams = 1:2,
    stem = TRUE)
Document-feature matrix of: 3 documents, 18 features (66.7% sparse).
3 x 18 sparse Matrix of class "dfmSparse"
       features
docs    quick brown fox the_quick quick_brown brown_fox like carrot i_like
  text1     1     1   1         1           1         1    0      0      0
  text2     0     0   0         0           0         0    1      1      1
  text3     0     0   0         0           0         0    0      0      0
       features
docs    like_carrot cat dog the_there there_that that_etc etc_cat cat_dog ETC.
  text1           0   0   0         0          0        0       0       0    0
  text2           1   0   0         0          0        0       0       0    0
  text3           0   1   1         1          1        1       1       1    1

为什么featnames（myDFM）包含多于一个或两个令牌的功能？

1 个答案: