在R中查找文本中最常出现的单词

时间:2016-05-18 06:38:09

标签: r n-gram

有人可以帮我解决如何使用R?

在文本中找到最常用的两个和三个单词

我的文字是......

text <- c("There is a difference between the common use of the term phrase and its technical use in linguistics. In common usage, a phrase is usually a group of words with some special idiomatic meaning or other significance, such as \"all rights reserved\", \"economical with the truth\", \"kick the bucket\", and the like. It may be a euphemism, a saying or proverb, a fixed expression, a figure of speech, etc. In grammatical analysis, particularly in theories of syntax, a phrase is any group of words, or sometimes a single word, which plays a particular role within the grammatical structure of a sentence. It does not have to have any special meaning or significance, or even exist anywhere outside of the sentence being analyzed, but it must function there as a complete grammatical unit. For example, in the sentence Yesterday I saw an orange bird with a white neck, the words an orange bird with a white neck form what is called a noun phrase, or a determiner phrase in some theories, which functions as the object of the sentence. Theorists of syntax differ in exactly what they regard as a phrase; however, it is usually required to be a constituent of a sentence, in that it must include all the dependents of the units that it contains. This means that some expressions that may be called phrases in everyday language are not phrases in the technical sense. For example, in the sentence I can't put up with Alex, the words put up with (meaning \'tolerate\') may be referred to in common language as a phrase (English expressions like this are frequently called phrasal verbs\ but technically they do not form a complete phrase, since they do not include Alex, which is the complement of the preposition with.")

5 个答案:

答案 0 :(得分:6)

您的文字是:

text <- c("There is a difference between the common use of the term phrase and its technical use in linguistics. In common usage, a phrase is usually a group of words with some special idiomatic meaning or other significance, such as \"all rights reserved\", \"economical with the truth\", \"kick the bucket\", and the like. It may be a euphemism, a saying or proverb, a fixed expression, a figure of speech, etc. In grammatical analysis, particularly in theories of syntax, a phrase is any group of words, or sometimes a single word, which plays a particular role within the grammatical structure of a sentence. It does not have to have any special meaning or significance, or even exist anywhere outside of the sentence being analyzed, but it must function there as a complete grammatical unit. For example, in the sentence Yesterday I saw an orange bird with a white neck, the words an orange bird with a white neck form what is called a noun phrase, or a determiner phrase in some theories, which functions as the object of the sentence. Theorists of syntax differ in exactly what they regard as a phrase; however, it is usually required to be a constituent of a sentence, in that it must include all the dependents of the units that it contains. This means that some expressions that may be called phrases in everyday language are not phrases in the technical sense. For example, in the sentence I can't put up with Alex, the words put up with (meaning \'tolerate\') may be referred to in common language as a phrase (English expressions like this are frequently called phrasal verbs\ but technically they do not form a complete phrase, since they do not include Alex, which is the complement of the preposition with.")

自然语言处理中,2个词的短语称为“ bi-gram ”,3个词的短语称为“ tri” -gram “,等等。通常,给定的n个单词组合称为“ n-gram ”。

首先,我们安装ngram包(在CRAN上可用)

# Install package "ngram"
install.packages("ngram")

然后,我们会找到最常见的双字和三字短语

library(ngram)

# To find all two-word phrases in the test "text":
ng2 <- ngram(text, n = 2)

# To find all three-word phrases in the test "text":
ng3 <- ngram(text, n = 3)

最后,我们将使用以下各种方法打印对象(ngrams):

print(ng, output="truncated")

print(ngram(x), output="full")

get.phrasetable(ng)

ngram::ngram_asweka(text, min=2, max=3)

我们也可以使用Markov Chains来唠叨新序列:

# if we are using ng2 (bi-gram)
lnth = 2 
babble(ng = ng2, genlen = lnth)

# if we are using ng3 (tri-gram)
lnth = 3  
babble(ng = ng3, genlen = lnth)

答案 1 :(得分:4)

最简单?

INSERT INTO yourTable (field1, field2, field3) VALUES ("information1", "", "");
UPDATE yourTable SET field2 = "information2", field3 = "information3" WHERE rowId = x;

答案 2 :(得分:3)

这里是5个最常用单词的简单基础R方法:

head(sort(table(strsplit(gsub("[[:punct:]]", "", text), " ")), decreasing = TRUE), 5)

#     a    the     of     in phrase 
#    21     18     12     10      8 

它返回的是一个带有频率计数的整数向量,向量的名称对应于计算的单词。

  • gsub("[[:punct:]]", "", text)删除标点,因为您不想计算,我猜
  • strsplit(gsub("[[:punct:]]", "", text), " ")将字符串拆分为空格
  • table()计算独特元素&#39;频率
  • sort(..., decreasing = TRUE)按递减顺序对其进行排序
  • head(..., 5)仅选择前5个最常用的字词

答案 3 :(得分:3)

我们可以拆分单词并使用表格来总结频率:

words <- strsplit(text, "[ ,.\\(\\)\"]")
sort(table(words, exclude = ""), decreasing = T)

答案 4 :(得分:3)

tidytext包让这类事情变得非常简单:

library(tidytext)
library(dplyr)

data_frame(text = text) %>% 
    unnest_tokens(word, text) %>%    # split words
    anti_join(stop_words) %>%    # take out "a", "an", "the", etc.
    count(word, sort = TRUE) %>%    # count occurrences

# Source: local data frame [73 x 2]
# 
#           word     n
#          (chr) (int)
# 1       phrase     8
# 2     sentence     6
# 3        words     4
# 4       called     3
# 5       common     3
# 6  grammatical     3
# 7      meaning     3
# 8         alex     2
# 9         bird     2
# 10    complete     2
# ..         ...   ...

如果问题是询问双字母和三元组的计数,tokenizers::tokenize_ngrams是有用的:

library(tokenizers)

tokenize_ngrams(text, n = 3L, n_min = 2L, simplify = TRUE) %>%    # tokenize bigrams and trigrams
    as_data_frame() %>%    # structure
    count(value, sort = TRUE)    # count

# Source: local data frame [531 x 2]
# 
#           value     n
#          (fctr) (int)
# 1        of the     5
# 2      a phrase     4
# 3  the sentence     4
# 4          as a     3
# 5        in the     3
# 6        may be     3
# 7    a complete     2
# 8   a phrase is     2
# 9    a sentence     2
# 10      a white     2
# ..          ...   ...