Question

我有一个数据集，其中包含列“ studentDetails”和“ studentId”。我在该数据集中训练了模型并保存了它。当我训练模型并保存训练后的模型，然后加载训练后的模型进行预测时，它成功地给了我输出。但是当我独立加载保存的模型并使用它进行预测时，它给我一个错误“ CountVectorizer-词汇表不适合”

这是我正在使用的代码：

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
import pickle
from sklearn.svm import LinearSVC 

X_train, X_test, y_train, y_test = train_test_split(df['studentDetails'], df['studentId'], random_state = 0)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts) 
classificationModel = LinearSVC().fit(X_train_tfidf, y_train) 
filename = 'finalized_model.sav'
pickle.dump(classificationModel, open(filename, 'wb'))

现在加载模型并进行预测：

from sklearn.feature_extraction.text import CountVectorizer
data_to_be_predicted="Alicia Scott is from United States"
filename = 'finalized_model.sav'
loaded_model = pickle.load(open(filename, 'rb'))
count_vect = CountVectorizer()
result = loaded_model.predict(count_vect.transform([data_to_be_predicted]))
print(result)

输出：

当我仅独立运行第二个代码片段时，它给我一个错误

错误：

CountVectorizer - Vocabulary wasn't fitted

我只是想知道，为什么在第二种情况下会出现错误，因为在获得正确结果的情况下，我没有在第一种情况下的任何地方重新定义count_vect = CountVectorizer（）。

Answer 1

第二个代码段的问题是您没有使用合适的CounVectorizer，它是一个新的CounVectorizer，因此没有安装。

我将建议您使用 fit 而不是 fit_transform ，这将为您返回已经拟合的CountVectorizer，然后您可以像对模型一样进行保存。

>

 from sklearn.model_selection import train_test_split
 from sklearn.feature_extraction.text import CountVectorizer
 from sklearn.feature_extraction.text import TfidfTransformer
 import pickle
 from sklearn.svm import LinearSVC 

 X_train, X_test, y_train, y_test = train_test_split(df['studentDetails'], df['studentId'], random_state = 0)
 count_vect = CountVectorizer().fit(X_train)
 X_train_counts = count_vect.transform(X_train)
 tfidf_transformer = TfidfTransformer().fit(X_train_counts)
 X_train_tfidf = tfidf_transformer.transform(X_train_counts) 
 classificationModel = LinearSVC().fit(X_train_tfidf, y_train) 
 filename = 'finalized_model.sav'
 pickle.dump(classificationModel, open(filename, 'wb'))
 pickle.dump(count_vect, open('count_vect, 'wb'))
 pickle.dump(tfidf_transformer, open('tfidf_transformer, 'wb'))

现在您可以在要进行预测时加载其中的3个：

from sklearn.feature_extraction.text import CountVectorizer
data_to_be_predicted="Alicia Scott is from United States"
filename = 'finalized_model.sav'
loaded_model = pickle.load(open(filename, 'rb'))
count_vect = pickle.load(open('count_vect', 'rb'))
result = loaded_model.predict(count_vect.transform([data_to_be_predicted]))
print(result)

运行训练有素的机器学习模型时出错

1 个答案: