使用NLTK

Question

使用NLTK为以下内容获取唯一字频的代码。

Seq Sentence
1 Let's try to be Good.
2 Being good doesn't make sense.
3 Good is always good.

输出：
{'good':3, 'let':1, 'try':1, 'to':1, 'be':1, 'being':1, 'doesn':1, 't':1, 'make':1, 'sense':1, 'is':1, 'always':1, '.':3, ''':2, 's':1}

Answer 1

试试这个

from collections import Counter
import pandas as pd
import nltk

sno = nltk.stem.SnowballStemmer('english')
s = "1   Let's try to be Good. 2   Being good doesn't make sense. 3   Good is always good."
s1 = s.split(' ')
d = pd.DataFrame(s1)
s2 = d[0].apply(lambda x: sno.stem(x))
counts =  Counter(s2)
print(counts)

输出将是：

Counter({'': 6, 'be': 2, 'good.': 2, 'good': 2, '1': 1, 'let': 1, 'tri': 1, 'to': 1, '2': 1, "doesn't": 1, 'make': 1, 'sense.': 1, '3': 1, 'is': 1, 'alway': 1})

Answer 2

如果您非常关注使用nltk，请参阅以下代码段

import nltk

text1 = '''Seq Sentence 
1   Let's try to be Good.
2   Being good doesn't make sense.
3   Good is always good.'''

words = nltk.tokenize.word_tokenize(text1)
fdist1 = nltk.FreqDist(words)

filtered_word_freq = dict((word, freq) for word, freq in fdist1.items() if not word.isdigit())

print(filtered_word_freq)

希望它有所帮助。

从以下地址引用了一些部分：

How to check if string input is a number?

Dropping specific words out of an NLTK distribution beyond stopwords