Question

我想从我的专栏＆＃34;推文＆＃34;中删除停用词。如何迭代每一行和每一项？

pos_tweets = [('I love this car', 'positive'),
    ('This view is amazing', 'positive'),
    ('I feel great this morning', 'positive'),
    ('I am so excited about the concert', 'positive'),
    ('He is my best friend', 'positive')]

test = pd.DataFrame(pos_tweets)
test.columns = ["tweet","class"]
test["tweet"] = test["tweet"].str.lower().str.split()

from nltk.corpus import stopwords
stop = stopwords.words('english')

Answer 1

我们可以从stopwords导入nltk.corpus，如下所示。有了它，我们用Python的列表理解和pandas.DataFrame.apply排除了停用词。

# Import stopwords with nltk.
from nltk.corpus import stopwords
stop = stopwords.words('english')

pos_tweets = [('I love this car', 'positive'),
    ('This view is amazing', 'positive'),
    ('I feel great this morning', 'positive'),
    ('I am so excited about the concert', 'positive'),
    ('He is my best friend', 'positive')]

test = pd.DataFrame(pos_tweets)
test.columns = ["tweet","class"]

# Exclude stopwords with Python's list comprehension and pandas.DataFrame.apply.
test['tweet_without_stopwords'] = test['tweet'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
print(test)
# Out[40]:
#                                tweet     class tweet_without_stopwords
# 0                    I love this car  positive              I love car
# 1               This view is amazing  positive       This view amazing
# 2          I feel great this morning  positive    I feel great morning
# 3  I am so excited about the concert  positive       I excited concert
# 4               He is my best friend  positive          He best friend

也可以使用pandas.Series.str.replace排除它。

pat = r'\b(?:{})\b'.format('|'.join(stop))
test['tweet_without_stopwords'] = test['tweet'].str.replace(pat, '')
test['tweet_without_stopwords'] = test['tweet_without_stopwords'].str.replace(r'\s+', ' ')
# Same results.
# 0              I love car
# 1       This view amazing
# 2    I feel great morning
# 3       I excited concert
# 4          He best friend

如果您无法导入停用词，可以按如下方式下载。

import nltk
nltk.download('stopwords')

另一种回答方法是从text.ENGLISH_STOP_WORDS导入sklearn.feature_extraction。

# Import stopwords with scikit-learn
from sklearn.feature_extraction import text
stop = text.ENGLISH_STOP_WORDS

请注意，scikit-learn停用词和nltk停用词中的单词数量不同。

Answer 2

使用列表理解

test['tweet'].apply(lambda x: [item for item in x if item not in stop])

返回：

0               [love, car]
1           [view, amazing]
2    [feel, great, morning]
3        [excited, concert]
4            [best, friend]

Answer 3

查看pd.DataFrame.replace（），它可能适合您：

In [42]: test.replace(to_replace='I', value="",regex=True)
Out[42]:
                              tweet     class
0                     love this car  positive
1              This view is amazing  positive
2           feel great this morning  positive
3   am so excited about the concert  positive
4              He is my best friend  positive

编辑：replace()将搜索字符串（甚至是子字符串）。对于例如如果rk是有时不期望的停用词，它会从work替换rk。

因此在这里使用regex：

for i in stop :
    test = test.replace(to_replace=r'\b%s\b'%i, value="",regex=True)

Answer 4

如果您想简单但又不想找回单词列表：

test["tweet"].apply(lambda words: ' '.join(word.lower() for word in words.split() if word not in stop))

将停止定义为OP。

from nltk.corpus import stopwords
stop = stopwords.words('english')

Python从pandas数据帧中删除停用词

4 个答案: