Question

我试图从文本文件中获取英文单词以获得简单的单词频率目标。如何过滤掉列表中的其他字符串？

from nltk.tokenize import word_tokenize
words = word_tokenize(message.replace('\n',' '))

print(words)

给出这样的输出：

['Amazon', 'b', 'maji_opai', 'am\\xcd\\x9ca\\xcd\\x89zon\\xe2\\x80\\xa6', '\\xcb\\x99\\xea\\x92\\xb3\\xe2\\x80\\x8b\\xcb\\x99', 'Amazon', "b'RT", 'WorkingGIrl', 'For', 'people', 'love', 'REAL', 'paperbacks', 'THE', 'PARIS', 'EFFECT', '10', 'right', 'https', '//', 'https', 'Amazon', "b'RT", 'AbsentiaSeries', 'ABSENTIA', 'IS', 'HERE', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', 'US', 'UK', 'Australia', 'Germany', 'Ireland', 'Italy', 'Netherlands', 'go', 'https', 'Amazon', "b'RT",

Answer 1

如果你有一个特定的单词列表，你可以使用一个简单的列表理解，如下所示：

{{1}}

如果你经常使用python，你应该深入研究列表理解，它会出现很多

Explanation of how list comprehension works?

http://www.pythonforbeginners.com/basics/list-comprehensions-in-python

Answer 2

nltk中有一个手工制作的推文标记器：

>>> from nltk.tokenize import TweetTokenizer
>>> tt = TweetTokenizer()
>>> tweet = 'Thanks to the historic TAX CUTS that I signed into law, your paychecks are going way UP, your taxes are going way DOWN, and America is once again OPEN FOR BUSINESS! #FakeNews'
>>> tt.tokenize(tweet)
['Thanks', 'to', 'the', 'historic', 'TAX', 'CUTS', 'that', 'I', 'signed', 'into', 'law', ',', 'your', 'paychecks', 'are', 'going', 'way', 'UP', ',', 'your', 'taxes', 'are', 'going', 'way', 'DOWN', ',', 'and', 'America', 'is', 'once', 'again', 'OPEN', 'FOR', 'BUSINESS', '!', '#FakeNews']

Word_tokenize推特数据

2 个答案: