使用count和tfidf作为scikit学习的功能

时间:2014-12-02 23:13:01

标签: python numpy nlp scikit-learn ml

我试图将两个计数和tfidf用作多项NB模型的特征。这是我的代码:

text = ["this is spam", "this isn't spam"]
labels = [0,1]
count_vectorizer = CountVectorizer(stop_words="english", min_df=3)

tf_transformer = TfidfTransformer(use_idf=True)
combined_features = FeatureUnion([("counts", self.count_vectorizer), ("tfidf", tf_transformer)]).fit(self.text)

classifier = MultinomialNB()
classifier.fit(combined_features, labels)

但我在FeatureUnion和tfidf上遇到错误:

TypeError: no supported conversion for types: (dtype('S18413'),)

知道为什么会这样吗?是不是可以同时将两个计数和tfidf作为特征?

1 个答案:

答案 0 :(得分:8)

该错误并非来自FeatureUnion,而是来自TfidfTransformer

你应该使用TfidfVectorizer而不是TfidfTransformer,变换器需要一个numpy数组作为输入而不是明文,因此TypeError

此外,您的测试句对于Tfidf测试来说太小了,所以尝试使用更大的测试句,这是一个例子:

from nltk.corpus import brown

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.pipeline import FeatureUnion
from sklearn.naive_bayes import MultinomialNB

# Let's get more text from NLTK
text = [" ".join(i) for i in brown.sents()[:100]]
# I'm just gonna assign random tags.
labels = ['yes']*50 + ['no']*50
count_vectorizer = CountVectorizer(stop_words="english", min_df=3)
tf_transformer = TfidfVectorizer(use_idf=True)
combined_features = FeatureUnion([("counts", count_vectorizer), ("tfidf", tf_transformer)]).fit_transform(text)
classifier = MultinomialNB()
classifier.fit(combined_features, labels)