这是我第一次运行k-fold交叉验证，我对从输出中看到的现象感到困惑。基本上，5倍交叉验证始终为模型8（Adaboost分类器）和模型9（梯度增强分类器）提供最高的准确度分数，如下所示。然而，当我使用20％的数据集作为测试数据单独运行这些ML模型时，模型7（随机森林分类器）总是根据混淆矩阵和AUC在所有5个模型中产生最高精度。我最初的期望是，如果我单独运行ML模型，具有高k倍交叉验证精度的ML模型也应该返回高精度。这似乎不是这种情况。有人可以向我解释为什么我会看到这种差异吗？
这些是我用来训练数据的ML模型：
model6 = DecisionTreeClassifier()
model7 = RandomForestClassifier(n_estimators=300)
model8 = AdaBoostClassifier(n_estimators=300)
model9 = GradientBoostingClassifier(n_estimators=300, learning_rate=1.0, max_depth=1, random_state=0)
model10 = KNeighborsClassifier(n_neighbors=5)
这是5折CV和个人ML模型的完整代码：
X_train, X_test, Y_train, Y_test = train_test_split(whole_data_input, whole_data_output, test_size=0.2)
X_train.reset_index(inplace=True)
#To remove the index column:
X_train.drop(['index'],axis=1,inplace=True)
X_test.reset_index(inplace=True)
#To remove the index column:
X_test.drop(['index'],axis=1,inplace=True)
Y_train.reset_index(inplace=True)
#To remove the index column:
Y_train.drop(['index'],axis=1,inplace=True)
Y_test.reset_index(inplace=True)
#To remove the index column:
Y_test.drop(['index'],axis=1,inplace=True)
warnings.filterwarnings('ignore')
model6 = DecisionTreeClassifier()
model7 = RandomForestClassifier(n_estimators=300)
model8 = AdaBoostClassifier(n_estimators=300)
model9 = GradientBoostingClassifier(n_estimators=300,
learning_rate=1.0,max_depth=1, random_state=0)
model10 = KNeighborsClassifier(n_neighbors=5)
model6.fit(X_train, Y_train)
model7.fit(X_train, Y_train)
model8.fit(X_train, Y_train)
model9.fit(X_train, Y_train)
model10.fit(X_train, Y_train)
# Perform 5-fold cross validation across different models:
#Here I am calling 'whole_data['label'] instead of the 'whole_data[['label']] I created earlier because cross validation only works with this data shape:
whole_data_output=whole_data['label']
print('THE FOLLOWING OUTPUT REPRESENT ACCURACIES OF 5-FOLD VALIDATIONS FROM VARIOUS ML MODELS:')
print()
scores = cross_val_score(model6, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model6, Decision Tree Classifier, is:' + str(scores))
print()
scores = cross_val_score(model7, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model7, Random Forest Classifier, is:' + str(scores))
print()
scores = cross_val_score(model8, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model8, Adaboost Classifier, is:' + str(scores))
print()
scores = cross_val_score(model9, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model9, Gradient Boosting Classifier, is:' + str(scores))
print()
scores = cross_val_score(model10, whole_data_input, whole_data_output, cv=5)
print('Cross-validated scores for model10, K Neighbors Classifier, is:' + str(scores))
print('THE FOLLOWING OUTPUT REPRESENT RESULTS FROM VARIOUS ML MODELS:')
print()
result6 = model6.predict(X_test)
result7 = model7.predict(X_test)
result8 = model8.predict(X_test)
result9 = model9.predict(X_test)
result10 = model10.predict(X_test)
from sklearn.metrics import classification_report
print('Classification report for model 6, decision tree classifier, is: ')
print(confusion_matrix(Y_test,result6))
print()
print(classification_report(Y_test,result6))
print()
print("Area under curve (auc) of model6 is: ", metrics.roc_auc_score(Y_test, result6))
print()
print('Classification report for model 7, random forest classifier, is: ')
print(confusion_matrix(Y_test,result7))
print()
print(classification_report(Y_test,result7))
print()
print("Area under curve (auc) of model7 is: ", metrics.roc_auc_score(Y_test, result7))
print()
print('Classification report for model 8, adaboost classifier, is: ')
print(confusion_matrix(Y_test,result8))
print()
print(classification_report(Y_test,result8))
print()
print("Area under curve (auc) of model8 is: ", metrics.roc_auc_score(Y_test, result8))
print()
print('Classification report for model 9, gradient boosting classifier, is: ')
print(confusion_matrix(Y_test,result9))
print()
print(classification_report(Y_test,result9))
print()
print("Area under curve (auc) of model9 is: ", metrics.roc_auc_score(Y_test, result9))
print()
print('Classification report for model 10, K neighbors classifier, is: ')
print(confusion_matrix(Y_test,result10))
print()
print(classification_report(Y_test,result10))
print()
print("Area under curve (auc) of model10 is: ", metrics.roc_auc_score(Y_test, result10))
print()
以下各种ML模型的5折交叉验证的输出表示准确度：
Cross-validated scores for model6, Decision Tree Classifier, is:[ 0.61364665 0.75754735 0.77046902]
Cross-validated scores for model7, Random Forest Classifier, is:[ 0.62463637 0.79326395 0.8073181 ]
Cross-validated scores for model8, Adaboost Classifier, is:[ 0.64916931 0.81960696 0.84196916]
Cross-validated scores for model9, Gradient Boosting Classifier, is:[ 0.64910466 0.82177258 0.83909235]
Cross-validated scores for model10, K Neighbors Classifier, is:[ 0.61180425 0.75412115 0.73012897]
以下输出代表的结果来自各种ML模型：
Classification report for model 6, decision tree classifier, is:
[[6975 1804]
[1893 7891]]
precision recall f1-score support
-1 0.79 0.79 0.79 8779
1 0.81 0.81 0.81 9784
avg / total 0.80 0.80 0.80 18563
Area under curve (auc) of model6 is: 0.800515237805
Classification report for model 7, random forest classifier, is:
[[6883 1896]
[1216 8568]]
precision recall f1-score support
-1 0.85 0.78 0.82 8779
1 0.82 0.88 0.85 9784
avg / total 0.83 0.83 0.83 18563
Area under curve (auc) of model7 is: 0.829872762782
Classification report for model 8, adaboost classifier, is:
[[5851 2928]
[ 891 8893]]
precision recall f1-score support
-1 0.87 0.67 0.75 8779
1 0.75 0.91 0.82 9784
avg / total 0.81 0.79 0.79 18563
Area under curve (auc) of model8 is: 0.787704885721
Classification report for model 9, gradient boosting classifier, is:
[[5905 2874]
[ 918 8866]]
precision recall f1-score support
-1 0.87 0.67 0.76 8779
1 0.76 0.91 0.82 9784
avg / total 0.81 0.80 0.79 18563
Area under curve (auc) of model9 is: 0.789400603089
Classification report for model 10, K neighbors classifier, is:
[[6467 2312]
[1666 8118]]
precision recall f1-score support
-1 0.80 0.74 0.76 8779
1 0.78 0.83 0.80 9784
avg / total 0.79 0.79 0.79 18563
Area under curve (auc) of model10 is: 0.783183129908
答案 0 :(得分：1)
尝试在cross_val_score中设置cv=StratifiedKFold(n_splits=5, shuffle=True)，看看它是否有所作为。我的理解是train_test_split将在类中随机抽样，但cross_val_score不会（默认情况下）。
您可以使用from sklearn.model_selection import StratifiedKFold