我正在使用NLTK和SKlearn测试情感分析模型。
Movie_reviews数据有" pos"和" neg"标签。为了训练分类器我正在使用" featuresets"。我正在使用交叉验证来训练数据和测试数据的准确性。然而,交叉验证总是远高于准确性。在下面的例子中,逻辑回归算法CV = 97(平均值),准确度= 70.我也测试了其他算法并且仍然交叉验证非常高。
我很确定我的交叉验证代码是不对的。
import nltk
import random
from nltk.corpus import movie_reviews
from sklearn import cross_validation
from nltk.classify.scikitlearn import SklearnClassifier
from sklearn.linear_model import LogisticRegression, SGDClassifier
documents = [(list(movie_reviews.words(fileid)), category)
for category in movie_reviews.categories()
for fileid in movie_reviews.fileids(category)]
random.shuffle(documents)
all_words = []
for w in movie_reviews.words():
all_words.append(w.lower())
all_words = nltk.FreqDist(all_words)
word_features = list(all_words.keys())[:3000]
def find_features(document):
words = set(document)
features = {}
for w in word_features:
features[w] = (w in words)
return features
featuresets = [(find_features(rev), category) for (rev, category) in documents]
training_set = featuresets[:1500]
testing_set = featuresets[1500:]
cv = cross_validation.KFold(len(training_set), n_folds=10, shuffle=True, random_state=None)
LogisticRegression_classifier = SklearnClassifier(LogisticRegression())
for traincv, testcv in cv:
classifier = LogisticRegression_classifier.train(training_set[traincv[0]:traincv[len(traincv)-1]])
print ('CV_accuracy:', nltk.classify.util.accuracy(classifier, training_set[testcv[0]:testcv[len(testcv)-1]]))
print("LogisticRegression_classifier accuracy percent:", (nltk.classify.accuracy(LogisticRegression_classifier, testing_set))*100)
答案 0 :(得分:1)
您正在使用 training_set [traincv [0]:traincv [len(traincv)-1]] ,这意味着范围从traincv [0]到traincv [len(traincv)-1]
在你的情况下,traincv [0]和testcv [0]将始终接近0并且traincv [len(traincv)-1]和testcv [len(testcv)-1]将接近1499.所以你是在进行N折验证时使用几乎相同的数据进行培训和测试。
在这里,您实际上需要使用traincv和testcv中的子集索引。
import numpy as np
training_set = np.array(training_set)
for traincv, testcv in cv:
classifier = LogisticRegression_classifier.train(training_set[traincv])
print ('CV_accuracy:', nltk.classify.util.accuracy(classifier, training_set[testcv]