Question

我是sklearn的新手。我正在使用Pipeline在Text挖掘问题中一起使用Vectorizer和Classifier。这是我的代码：

def create_ngram_model():
tfidf_ngrams = TfidfVectorizer(ngram_range=(1, 3),
analyzer="word", binary=False)
clf = GaussianNB()
pipeline = Pipeline([('vect', tfidf_ngrams), ('clf', clf)])
return pipeline


def get_trains():
    data=open('../cleaning data/cleaning the sentences/cleaned_comments.csv','r').readlines()[1:]
    lines=len(data)
    features_train=[]
    labels_train=[]
    for i in range(lines):
        l=data[i].split(',')
        labels_train+=[int(l[0])]
        a=l[2]
        features_train+=[a]
    return features_train,labels_train

def train_model(clf_factory,features_train,labels_train):
    features_train,labels_train=get_trains()
    features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(features_train, labels_train, test_size=0.1, random_state=42)
    clf=clf_factory()
    clf.fit(features_train,labels_train)
    pred = clf.predict(features_test)
    accuracy = accuracy_score(pred,labels_test)
    return accuracy

X,Y=get_trains()
print train_model(create_ngram_model,X,Y)

get_trains（）返回的功能是字符串。我收到了这个错误。

clf.fit(features_train,labels_train)
  File "C:\Python27\lib\site-packages\sklearn\pipeline.py", line 130, in fit
    self.steps[-1][-1].fit(Xt, y, **fit_params)
  File "C:\Python27\lib\site-packages\sklearn\naive_bayes.py", line 149, in fit
    X, y = check_arrays(X, y, sparse_format='dense')
  File "C:\Python27\lib\site-packages\sklearn\utils\validation.py", line 263, in check_arrays
    raise TypeError('A sparse matrix was passed, but dense '
TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.

我多次遇到过这个错误。然后，我只是将功能更改为features_transformed.toarray（）但是，因为在这里，我使用管道我无法这样做，因为转换的功能会自动返回。我也尝试创建一个返回features_transformed.toarray（）的新类，但是也抛出了同样的错误。我已经搜索了很多但没有得到它。请帮忙!!

Answer 1

有两个选项：

使用与稀疏数据兼容的分类器。例如，文档说Bernoulli Naive Bayes和Multinomial Naive Bayes支持fit的稀疏输入。

将“densifier”添加到Pipeline。显然，你弄错了，这个对我有用（当我需要在途中密集我的稀疏数据时）：

class Densifier(object):
    def fit(self, X, y=None):
        pass
    def fit_transform(self, X, y=None):
        return self.transform(X)
    def transform(self, X, y=None):
        return X.toarray()

确保在分类器之前加入管道。

sklearn中的Pipelining问题

1 个答案: