Question

真的有点简单的问题，但我似乎无法破解它。我有一个按以下方式格式化的字符串：

["category1",("data","data","data")]
["category2", ("data","data","data")]

我调用不同类别的帖子，我想从数据部分获取最常用的字词。所以我试过了：

from nltk.tokenize import wordpunct_tokenize
from collections import defaultdict
freq_dict = defaultdict(int)

for cat, text2 in posts:
   tokens = wordpunct_tokenize(text2)
   for token in tokens:
       if token in freq_dict:
           freq_dict[token] += 1
       else:
           freq_dict[token] = 1
   top = sorted(freq_dict, key=freq_dict.get, reverse=True)
   top = top[:50]
   print top

但是，这会给我字符串中的PER字句。

我需要一个通用的顶级单词列表但是，如果我从for循环中取出打印顶部，它只会给我最后一篇文章的结果有没有人有想法？

Answer 1

这是范围问题。此外，您无需初始化defaultdict的元素，因此这简化了您的代码：

试试这样：

posts = [["category1",("data1 data2 data3")],["category2", ("data1 data3 data5")]]

from nltk.tokenize import wordpunct_tokenize
from collections import defaultdict
freq_dict = defaultdict(int)

for cat, text2 in posts:
   tokens = wordpunct_tokenize(text2)
   for token in tokens:
      freq_dict[token] += 1

top = sorted(freq_dict, key=freq_dict.get, reverse=True)
top = top[:50]
print top

正如预期的那样，输出

['data1', 'data3', 'data5', 'data2']

结果是

。

如果真的有类似

的内容

posts = [["category1",("data1","data2","data3")],["category2", ("data1","data3","data5")]]

作为输入，您不需要wordpunct_tokenize()，因为输入数据已经被标记化。然后，以下将起作用：

posts = [["category1",("data1","data2","data3")],["category2", ("data1","data3","data5")]]

from collections import defaultdict
freq_dict = defaultdict(int)

for cat, tokens in posts:
   for token in tokens:
      freq_dict[token] += 1

top = sorted(freq_dict, key=freq_dict.get, reverse=True)
top = top[:50]
print top

并输出预期结果：

['data1', 'data3', 'data5', 'data2']

Answer 2

为什么不使用Counter？

In [30]: from collections import Counter

In [31]: data=["category1",("data","data","data")]

In [32]: Counter(data[1])
Out[32]: Counter({'data': 3})

In [33]: Counter(data[1]).most_common()
Out[33]: [('data', 3)]

Answer 3

from itertools import chain
from collections import Counter
from nltk.tokenize import wordpunct_tokenize
texts=["a quick brown car", "a fast yellow rose", "a quick night rider", "a yellow officer"]
print Counter(chain.from_iterable(wordpunct_tokenize(x) for x in texts)).most_common(3)

输出：

[('a', 4), ('yellow', 2), ('quick', 2)]

正如您在Counter.most_common的文档中所看到的，返回的列表已经过排序。

要使用您的代码，您可以

texts = (x[1] for x in posts)

或者你可以做

... wordpunct_tokenize(x[1]) for x in texts ...

如果你的帖子实际上是这样的：

posts=[("category1",["a quick brown car", "a fast yellow rose"]), ("category2",["a quick night rider", "a yellow officer"])]

你可以摆脱类别：

texts = list(chain.from_iterable(x[1] for x in posts))

（texts将为['a quick brown car', 'a fast yellow rose', 'a quick night rider', 'a yellow officer']）

然后，您可以在此答案顶部的代码段中使用它。

Answer 4

只需更改代码以允许处理帖子，然后获取热门词语：

from nltk.tokenize import wordpunct_tokenize
from collections import defaultdict

freq_dict = defaultdict(int)

for cat, text2 in posts:
   tokens = wordpunct_tokenize(text2)
   for token in tokens:
       freq_dict[token] += 1
# get top after all posts have been processed.
top = sorted(freq_dict, key=freq_dict.get, reverse=True)
top = top[:50]
print top

如何在多个单独的文本中找到最常用的单词？

4 个答案: