我有一个包含n-gram的词汇表,如下所示。
myvocabulary = ['tim tam', 'jam', 'fresh milk', 'chocolates', 'biscuit pudding']
我想用这些词来计算TF-IDF值。
我还有一个语料库词典如下(键=食谱编号,值=食谱)。
corpus = {1: "making chocolates biscuit pudding easy first get your favourite biscuit chocolates", 2: "tim tam drink new recipe that yummy and tasty more thicker than typical milkshake that uses normal chocolates", 3: "making chocolates drink different way using fresh milk egg"}
我目前正在使用以下代码。
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(vocabulary = myvocabulary, stop_words = 'english')
tfs = tfidf.fit_transform(corpus.values())
现在我在corpus
中打印令牌或n-gram的配方1以及tF-IDF值,如下所示。
feature_names = tfidf.get_feature_names()
doc = 0
feature_index = tfs[doc,:].nonzero()[1]
tfidf_scores = zip(feature_index, [tfs[doc, x] for x in feature_index])
for w, s in [(feature_names[i], s) for (i, s) in tfidf_scores]:
print(w, s)
我得到的结果是chocolates 1.0
。但是,在计算TF-IDF值时,我的代码不会检测到诸如biscuit pudding
之类的n-gram(bigrams)。请告诉我代码错误的地方。
我希望使用myvocabulary
中的食谱文档获取corpus
项的TD-IDF矩阵。换句话说,矩阵的行代表myvocabulary
,矩阵的列代表我的corpus
的配方文档。请帮帮我。
答案 0 :(得分:10)
尝试增加ngram_range
中的TfidfVectorizer
:
tfidf = TfidfVectorizer(vocabulary = myvocabulary, stop_words = 'english', ngram_range=(1,2))
编辑: TfidfVectorizer
的输出是稀疏格式的TF-IDF矩阵(实际上是以您寻找的格式转置它)。你可以打印出它的内容,例如像这样:
feature_names = tfidf.get_feature_names()
corpus_index = [n for n in corpus]
rows, cols = tfs.nonzero()
for row, col in zip(rows, cols):
print((feature_names[col], corpus_index[row]), tfs[row, col])
应该产生
('biscuit pudding', 1) 0.646128915046
('chocolates', 1) 0.763228291628
('chocolates', 2) 0.508542320378
('tim tam', 2) 0.861036995944
('chocolates', 3) 0.508542320378
('fresh milk', 3) 0.861036995944
如果矩阵不大,则可能更容易以密集形式检查它。 Pandas
使这非常方便:
import pandas as pd
df = pd.DataFrame(tfs.T.todense(), index=feature_names, columns=corpus_index)
print(df)
这导致
1 2 3
tim tam 0.000000 0.861037 0.000000
jam 0.000000 0.000000 0.000000
fresh milk 0.000000 0.000000 0.861037
chocolates 0.763228 0.508542 0.508542
biscuit pudding 0.646129 0.000000 0.000000