使用sklearn.datasets.load_files在Scikit-Learn中加载数据时保留示例

时间:2017-06-02 16:21:47

标签: python scikit-learn text-classification

我正在尝试使用Scikit-learn的简单Naive Bayes。

基本上,我有两个文件夹,分别命名为Cat A和Cat B,每个文件夹包含大约1,500个文本文件。

我正在加载这些文件以便像这样训练分类器:

# Declare the categories
categories = ['CatA', 'CatB']

# Load the dataset
docs_to_train = sklearn.datasets.load_files("/Users/dh/Documents/Development/Python/Test_Data", description=None, categories=categories, load_content=True, shuffle=True, encoding='utf-8', decode_error='strict', random_state=0)

我用短文本字符串测试分类器,例如

docs_new = ['This is test string 1.', 'This is test string 2.', 'This is test string 3.']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)

predicted = clf.predict(X_new_tfidf)

for doc, category in zip(docs_new, predicted):
    print('%r => %s' % (doc, docs_to_train.target_names[category]))

一切都按照应有的方式运作,但我真正想要做的是测试一些与训练数据非常相似的数据的分类器。理想情况下,我想在数据中创建一个保留样本,使用训练分类器,然后与之交叉验证。

我想我可以将每个训练数据集中的500多个文档移动到不同的文件夹中,但我想知道是否有更好的方法来创建保留样本?

documentation似乎没有提供此指导。

完整的代码如下。

import sklearn
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
import numpy as np
from sklearn import datasets
from pprint import pprint

# Declare the categories
categories = ['CatA', 'CatB']

# Load the dataset
docs_to_train = sklearn.datasets.load_files("/Users/dh/Documents/Development/Python/Test_Data", description=None, categories=categories, load_content=True, shuffle=True, encoding='utf-8', decode_error='strict', random_state=0)

print len(docs_to_train.data)

# Vectorise the dataset

count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(docs_to_train.data)

# Fit the estimator and transform the vector to tf-idf

tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)
X_train_tf.shape

tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
print X_train_tfidf.shape

clf = MultinomialNB().fit(X_train_tfidf, docs_to_train.target)

docs_new = ['I am test string 1.', 'I am test string 2', 'I am test string 3']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)

predicted = clf.predict(X_new_tfidf)

for doc, category in zip(docs_new, predicted):
    print('%r => %s' % (doc, docs_to_train.target_names[category])) 

1 个答案:

答案 0 :(得分:1)

您正在寻找的内容被称为"列车测试分割。"

使用sklearn.model_selection.train_test_split

from sklearn.model_selection import train_test_split

train_X, test_X, train_y, test_y = train_test_split(docs_to_train.data, 
                               docs_to_train.target,
                               test_size = 500)