将列添加到Tfidf矩阵

时间:2015-04-15 14:42:44

标签: python scipy scikit-learn nltk

我想使用单词以及一些其他功能(例如,有链接)在文本上构建分类模型

tweets = ['this tweet has a link htt://link','this one does not','this one does http://link.net']

我使用sklearn来获取文本数据的稀疏矩阵

tfidf_vectorizer = TfidfVectorizer(max_df=0.90, max_features=200000, min_df=0.1, stop_words='english', use_idf=True, ntlk.tokenize,ngram_range=(1,2))

tfidf_matrix = tfidf_vectorizer.fit_transform(tweets)

我想为其添加列以支持文本数据的其他功能。我试过了:

import scipy as sc

all_data = sc.hstack((tfidf_matrix, [1,0,1]))

这给了我这样的数据:

array([ <3x8 sparse matrix of type '<type 'numpy.float64'>' with 10 stored elements in Compressed Sparse Row format>, 1, 1, 0], dtype=object)

当我将此数据框提供给模型时:

`from sklearn.naive_bayes import MultinomialNB
 clf = MultinomialNB().fit(all_data, y)` 

我收到了追溯错误:

`Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "C:\Anaconda\lib\site- packages\spyderlib\widgets\externalshell\sitecustomize.py", line 580, in   runfile
 execfile(filename, namespace)
 File "C:/Users/c/Desktop/features.py", line 157, in <module>
 clf = MultinomialNB().fit(all_data, y)
File "C:\Anaconda\lib\site-packages\sklearn\naive_bayes.py", line 302, in  fit
_, n_features = X.shape

ValueError:需要多于1个值来解包`

编辑:数据的形状

`tfidf_matrix.shape
 (100, 2)
 all_data.shape
 (100L,)`

我可以将列直接追加到稀疏矩阵吗?如果没有,我应该如何将数据转换为可以支持此格式的格式?我担心除了稀疏矩阵之外的其他东西会增加内存占用。

3 个答案:

答案 0 :(得分:11)

“我可以将直接列添加到稀疏矩阵吗?” - 是的你可能应该这样做,因为解包(使用todensetoarray)很容易导致大型语料库中的内存爆炸。

使用scipy.sparse.hstack

import numpy as np
import scipy as sp
from sklearn.feature_extraction.text import TfidfVectorizer

tweets = ['this tweet has a link htt://link','this one does not','this one does http://link.net']
tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(tweets)
print tfidf_matrix.shape

(3,10)

new_column = np.array([[1],[0],[1]])
print new_column.shape

(3,1)

final = sp.sparse.hstack((tfidf_matrix, new_column))
print final.shape

(3,11)

答案 1 :(得分:1)

将稀疏矩阵转换为密集矩阵

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
tweets = ['this tweet has a link htt://link','this one does not','this one does http://link.net']

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(tweets)
dense = tfidf_matrix.todense()
print dense.shape

newCol = [[1],[0],[1]]

allData = np.append(dense, newCol, 1)

print allData.shape

(3L,10L)

(3L,11L)

答案 2 :(得分:0)

这是正确的格式:

all_data = sc.hstack([tfidf_matrix, sc.csr_matrix([1,0,1]).T], 'csr')