Question

我正在使用以下代码：

import pandas as pd
import numpy as np
from nltk.tokenize import word_tokenize
import re

使用TFIDF向量化

from sklearn.feature_extraction.text import TfidfVectorizer
tv=TfidfVectorizer(max_df=0.5,min_df=2,stop_words='english')

加载数据文件

df=pd.read_json('train.json',orient='columns')
test_df=pd.read_json('test.json',orient='columns')

df['seperated_ingredients'] = df['ingredients'].apply(','.join)
test_df['seperated_ingredients'] = test_df['ingredients'].apply(','.join)

df['seperated_ingredients']=df['seperated_ingredients'].str.lower()
test_df['seperated_ingredients']=test_df['seperated_ingredients'].str.lower()

cuisines={'thai':0,'vietnamese':1,'spanish':2,'southern_us':3,'russian':4,'moroccan':5,'mexican':6,'korean':7,'japanese':8,'jamaican':9,'italian':10,'irish':11,'indian':12,'greek':13,'french':14,'filipino':15,'chinese':16,'cajun_creole':17,'british':18,'brazilian':19 }
df.cuisine= [cuisines[item] for item in df.cuisine]

进行预处理

ho=df['seperated_ingredients']
ho=ho.replace(r'#([^\s]+)', r'\1', regex=True)
ho=ho.replace('\'"',regex=True)

ho=tv.fit_transform(ho)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(ho,df['cuisine'],random_state=0)


from sklearn.linear_model import LogisticRegression
clf= LogisticRegression(penalty='l1')
clf.fit(X_train, y_train)
clf.score(X_test,y_test)

from sklearn.linear_model import LogisticRegression
clf1= LogisticRegression(penalty='l1')
clf1.fit(ho,df['cuisine'])

hs=test_df['seperated_ingredients']

hs=hs.replace(r'#([^\s]+)', r'\1', regex=True)
hs=hs.replace('\'"',regex=True)
hs=tv.fit_transform(hs)

ss=clf1.predict(hs) # this line is giving error.

在预测时得到上述错误。有人知道我在做什么错吗？

Answer 1

您不应改装tfidf-vectorizer，而应使用具有相同词汇形状的相同矢量化器对测试数据进行编码。 docs中有方法描述：

fit_transform(raw_documents, y=None)
  Learn vocabulary and idf, return term-document matrix.
  This is equivalent to fit followed by transform, but more efficiently implemented.

transform(raw_documents, copy=True)
  Transform documents to document-term matrix.
  Uses the vocabulary and document frequencies (df) learned by fit (or fit_transform).

您有ValueError: X has 1709 features per sample; expecting 2444，因为矢量化器已重新装入测试数据，并且创建了新词汇，因此测试数据被编码为其他形状的数组。只需使用print(len(tv.vocabulary_))检查第二次fit_transform前后的词汇量即可。另外，tf-idf词汇表可能在改编过程中进行了重新排序。

ho=df['seperated_ingredients']
ho=ho.replace(r'#([^\s]+)', r'\1', regex=True)
ho=ho.replace('\'"',regex=True)
ho=tv.fit_transform(ho)

然后使用预先训练的tf-idf矢量化器对具有转换功能的数据进行编码：

hs=test_df['seperated_ingredients']
hs=hs.replace(r'#([^\s]+)', r'\1', regex=True)
hs=hs.replace('\'"',regex=True)
hs=tv.transform(hs)

使用相同的词汇进行转换，因此输出数组具有正确的形状。

ValueError：X每个样本具有1709个功能；期待2444

1 个答案: