Question

我正在预处理文本数据，（确切地说是推特数据），但每当我应用NLTK词干分析器时，我都会得到一个NoneTypes列表。我无法弄清楚为什么会发生这种情况，我不知道如何解决它。

这是我的文本数据处理过程的方式：

处理前：

In [10]:
undefined



import pandas as pd
import numpy as np
import glob
import os
import nltk
dir = "C:\Users\Anonymous\Desktop\KAGA FOLDER\Hashtags"
train = np.array(pd.read_csv(os.path.join(dir,"train.csv")))[:,1]
def clean_the_text(data):
    alist = []
    data = nltk.word_tokenize(data)
    for j in data:
        alist.append(j.rstrip('\n'))
    alist = " ".join(alist)
    return alist

def stemmer(data):
    stemmer = nltk.stem.PorterStemmer()
    new_list = []
    new_list = [new_list.append(stemmer.stem(word)) for word in data]
    return new_list
def loop_data(data):
    for i in range(len(data)):
        data[i] = clean_the_text(data[i])
    return data
train


Out[10]:
array(['Jazz for a Rainy Afternoon:  {link}',
       'RT: @mention: I love rainy days.',
       'Good Morning Chicago! Time to kick the Windy City in the nuts and head back West!',
       ...,
       'OMG #WeatherForecast for tomm 80 degrees & Sunny &lt;=== #NeedThat #Philly #iMustSeeItToBelieveIt yo',
       "@mention Oh no! We had cold weather early in the week, but now it's getting warmer! Hoping the rain holds out to Saturday!",
       'North Cascades Hwy to reopen Wed.: quite late after a long, deep winter. Only had to clear snow 75 ft deep {link}'], dtype=object)

标记和清理文字后

train = loop_data(train)

In [12]:
undefined



train
Out[12]:
array(['Jazz for a Rainy Afternoon : { link }',
       'RT : @ mention : I love rainy days .',
       'Good Morning Chicago ! Time to kick the Windy City in the nuts and head back West !',
       ...,
       'OMG # WeatherForecast for tomm 80 degrees & Sunny & lt ; === # NeedThat # Philly # iMustSeeItToBelieveIt yo',
       "@ mention Oh no ! We had cold weather early in the week , but now it 's getting warmer ! Hoping the rain holds out to Saturday !",
       'North Cascades Hwy to reopen Wed. : quite late after a long , deep winter. Only had to clear snow 75 ft deep { link }'], dtype=object)

最终在阻止之后：

In [13]:
undefined



train = stemmer(train)
train
Out[13]:
[None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,

Answer 1

问题在于：new_list = [new_list.append(stemmer.stem(word)) for word in data]。它应该是

new_list = [stemmer.stem(word) for word in data]
# or 
# new_data = map(stemmer.stem, data) # returns a map object

new_list正在追加len（数据）次，然后将其设置为列表推导语句中的新列表，其中包含new_list.append的len（数据）结果，即无。

NLTK词干分析器返回NoneTypes列表

1 个答案: