Question

所以我创建了一个分类器，用于区分欺诈性消息和真实消息。代码片段如下：

# Import training set as DataFrame from CSV
dataset = pd.read_csv('data.csv', sep=',')
class_names = { 1: 'no-flag', 2: 'flag' }

# Separate training data to message, class pairs
X_train, y_train = dataset.iloc[:,0], dataset.iloc[:, 1]

messages = pd.read_csv('messages.csv', header=None)
X_predict = messages.iloc[:,0]

print "TRAIN:\n"
print type(X_train)
print "PREDICT:\n"
print type(X_predict)

# Vectorise text data
vect = TfidfVectorizer(ngram_range=(1, 2), lowercase=True, preprocessor=sanitise_message)
X_train_tfidf = vect.fit_transform(X_train)
X_predict_tfidf = vect.transform(X_predict)

我过去常常在训练集上运行十倍交叉验证，使用：

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=1)

过去工作正常。现在我想将整个训练集用作训练数据，并预测未分类的数据。但是，对X_predict_tfidf = vect.transform(X_predict)的调用会引发错误，如下所示：

Traceback (most recent call last):
File "post-test.py", line 3, in <module>
classify()
File "/Users/user/Documents/MyTutor/mi_datawarehouse/classifier.py", line 90, in classify
X_predict_tfidf = vect.transform(X_predict)
File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1409, in transform
X = super(TfidfVectorizer, self).transform(raw_documents)
File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 923, in transform
_, X = self._count_vocab(raw_documents, fixed_vocab=True)
File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 792, in _count_vocab
for feature in analyze(doc):
File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 266, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 119, in decode
raise ValueError("np.nan is an invalid document, expected byte or "
ValueError: np.nan is an invalid document, expected byte or unicode string.

有趣的是，X_train和X_predict的类型是相同的：

TRAIN:
<class 'pandas.core.series.Series'>
PREDICT:
<class 'pandas.core.series.Series'>

我做错了什么？我一直在疯狂，因为我到处寻找，包括scikit-learn文档。

注意：这不是a similar question的重复，我已经尝试过该问题中的所有内容，但没有任何效果。数据结构和问题略有不同。

Answer 1

快速修复可能是删除NaN。试试messages.dropna()

Pandas DataFrame与sci-kit fit_transform（）函数

1 个答案: