Question

我有我的停用词列表，我想用它来删除文本中的特定短语：

   #dummy text
    df2 <- c("hi my name is Ann and code code all the time! However not after that I would like")

mystopwords <- c("hi", "code code", "not after that")

我使用此选项：

myDfm <- df2 %>%
  tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
  tokens_remove(pattern = c(stopwords(source = "smart"), mystopwords)) %>%
  tokens_wordstem() %>%
  tokens_ngrams(n = c(1, 3)) %>%
  dfm()

但是当我检查bigram或trigram的频率时，它们并没有被删除。

语法有什么问题吗？

Answer 1

使用停止短语列表时，可以通过使用phrase()函数来实现。

它是这样的：

library(quanteda)
df2 <- c("hi my name is Ann and code code all the time! However not after that I would like")

mystopwords <- c("hi", "code code", "not after that")

df2 %>% tokens %>% 
  tokens_remove(pattern = phrase(mystopwords), valuetype = 'fixed')

## tokens from 1 document.
## text1 :
##  [1] "my"      "name"    "is"      "Ann"     "and"     "all"     "the"     "time"    "!"       "However" "I"       "would"  
## [13] "like"

您可以在 quanteda 中获得有关如何使用多字表达式的详细信息： https://quanteda.io/articles/pkgdown/examples/phrase.html

使用Quanteda删除自定义停用词和短语

1 个答案: