为collocationbigramfinder NLTK删除停用词和标记化

Question

我一直收到此错误

sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

当我尝试运行此脚本时。不确定有什么问题。我基本上是从文本文件中读取，过滤掉停用词并使用NLTK对它们进行标记。

import nltk
from nltk.collocations import *
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

stopset = set(stopwords.words('english'))
bigram_measures = nltk.collocations.BigramAssocMeasures()
trigram_measures = nltk.collocations.TrigramAssocMeasures()


text_file=open('sentiment_test.txt', 'r')
lines=text_file.readlines()
filtered_words = [w for w in lines if not w in stopwords.words('english')]
print filtered_words
tokens=word_tokenize(str(filtered_words)
print tokens
finder = BigramCollocationFinder.from_words(tokens)

非常感谢任何帮助。

Answer 1

我假设sentiment_test.txt只是纯文本，而不是特定的格式。 您正在尝试过滤行而不是单词。您应该首先标记然后过滤掉停用词。

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

stopset = set(stopwords.words('english'))

with open('sentiment_test.txt', 'r') as text_file:
    text = text_file.read()
    tokens=word_tokenize(str(text))
    tokens = [w for w in tokens if not w in stopset]
    print tokens

希望这有帮助。