Question

我正在尝试删除停用词并尝试以下操作：

tokenizer = RegexpTokenizer(r'\w+')
tokenized = data['data_column'].apply(tokenizer.tokenize)
tokenized

标记化后的输出

0    [ANOTHER,SAMPLE,AS,OUTPUT,MSG...
1    [A,SAMPLE,TEXT,FOR,ILLUSTRATION...
Name: data_column, dtype: object

我尝试使用以下内容删除停用词：

stop_words = set(stopwords.words('english'))
filtered_sentence = [w for w in tokenized if not w in stop_words]
filtered_sentence = []
 for w in tokenized:
    if w not in stop_words:
        filtered_sentence.append(w)

我收到错误：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-272-d4a699384ffc> in <module>()
      2 stop_words = set(stopwords.words('english'))
      3 
----> 4 filtered_sentence = [w for w in tokenized if not w in stop_words]
      5 
      6 filtered_sentence = []

TypeError: unhashable type: 'list'

Answer 1

您需要.apply()从列表中过滤列表，因为语料库中包含您需要在搜索之前使用.lower（）的小写单词，即

stop_words = set(stopwords.words('english'))
filtered_sentence = tokenized.apply(lambda x : [w for w in x if w.lower() not in stop_words])

样品运行

from nltk.corpus import stopwords
stop = set(stopwords.words('english'))

df = pd.DataFrame({'words': [['A','SAMPLE','AS','OUTPUT','MSG']]})
df['words'].apply(lambda x : [i for i in x if not i.lower() in stop])

0    [SAMPLE, OUTPUT, MSG]
Name: words, dtype: object

Nltk：消除列表清单中的停用词

1 个答案: