我正在尝试使用Scikit-learn的简单Naive Bayes。
基本上,我有两个文件夹,分别命名为Cat A和Cat B,每个文件夹包含大约1,500个文本文件。
我正在加载这些文件以便像这样训练分类器:
# Declare the categories
categories = ['CatA', 'CatB']
# Load the dataset
docs_to_train = sklearn.datasets.load_files("/Users/dh/Documents/Development/Python/Test_Data", description=None, categories=categories, load_content=True, shuffle=True, encoding='utf-8', decode_error='strict', random_state=0)
我用短文本字符串测试分类器,例如
docs_new = ['This is test string 1.', 'This is test string 2.', 'This is test string 3.']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)
predicted = clf.predict(X_new_tfidf)
for doc, category in zip(docs_new, predicted):
print('%r => %s' % (doc, docs_to_train.target_names[category]))
一切都按照应有的方式运作,但我真正想要做的是测试一些与训练数据非常相似的数据的分类器。理想情况下,我想在数据中创建一个保留样本,使用训练分类器,然后与之交叉验证。
我想我可以将每个训练数据集中的500多个文档移动到不同的文件夹中,但我想知道是否有更好的方法来创建保留样本?
documentation似乎没有提供此指导。
完整的代码如下。
import sklearn
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
import numpy as np
from sklearn import datasets
from pprint import pprint
# Declare the categories
categories = ['CatA', 'CatB']
# Load the dataset
docs_to_train = sklearn.datasets.load_files("/Users/dh/Documents/Development/Python/Test_Data", description=None, categories=categories, load_content=True, shuffle=True, encoding='utf-8', decode_error='strict', random_state=0)
print len(docs_to_train.data)
# Vectorise the dataset
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(docs_to_train.data)
# Fit the estimator and transform the vector to tf-idf
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)
X_train_tf.shape
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
print X_train_tfidf.shape
clf = MultinomialNB().fit(X_train_tfidf, docs_to_train.target)
docs_new = ['I am test string 1.', 'I am test string 2', 'I am test string 3']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)
predicted = clf.predict(X_new_tfidf)
for doc, category in zip(docs_new, predicted):
print('%r => %s' % (doc, docs_to_train.target_names[category]))
答案 0 :(得分:1)
您正在寻找的内容被称为"列车测试分割。"
使用sklearn.model_selection.train_test_split
:
from sklearn.model_selection import train_test_split
train_X, test_X, train_y, test_y = train_test_split(docs_to_train.data,
docs_to_train.target,
test_size = 500)