Keras模型的交叉验证

Question

我正在尝试对k-nn分类器进行交叉验证，但我对以下以下两种方法中的哪种正确进行交叉验证感到困惑。

training_scores = defaultdict(list)
validation_f1_scores = defaultdict(list)
validation_precision_scores = defaultdict(list)
validation_recall_scores = defaultdict(list)
validation_scores = defaultdict(list)

def model_1(seed, X, Y):
    np.random.seed(seed)
    scoring = ['accuracy', 'f1_macro', 'precision_macro', 'recall_macro']
    model = KNeighborsClassifier(n_neighbors=13)

    kfold = StratifiedKFold(n_splits=2, shuffle=True, random_state=seed)
    scores = model_selection.cross_validate(model, X, Y, cv=kfold, scoring=scoring, return_train_score=True)
    print(scores['train_accuracy'])
    training_scores['KNeighbour'].append(scores['train_accuracy'])
    print(scores['test_f1_macro'])
    validation_f1_scores['KNeighbour'].append(scores['test_f1_macro'])
    print(scores['test_precision_macro'])
    validation_precision_scores['KNeighbour'].append(scores['test_precision_macro'])
    print(scores['test_recall_macro'])
    validation_recall_scores['KNeighbour'].append(scores['test_recall_macro'])
    print(scores['test_accuracy'])
    validation_scores['KNeighbour'].append(scores['test_accuracy'])

    print(np.mean(training_scores['KNeighbour']))
    print(np.std(training_scores['KNeighbour']))
    #rest of print statments

第二个模型中的for循环似乎是多余的。

def model_2(seed, X, Y):
    np.random.seed(seed)
    scoring = ['accuracy', 'f1_macro', 'precision_macro', 'recall_macro']
    model = KNeighborsClassifier(n_neighbors=13)

    kfold = StratifiedKFold(n_splits=2, shuffle=True, random_state=seed)
    for train, test in kfold.split(X, Y):
        scores = model_selection.cross_validate(model, X[train], Y[train], cv=kfold, scoring=scoring, return_train_score=True)
        print(scores['train_accuracy'])
        training_scores['KNeighbour'].append(scores['train_accuracy'])
        print(scores['test_f1_macro'])
        validation_f1_scores['KNeighbour'].append(scores['test_f1_macro'])
        print(scores['test_precision_macro'])
        validation_precision_scores['KNeighbour'].append(scores['test_precision_macro'])
        print(scores['test_recall_macro'])
        validation_recall_scores['KNeighbour'].append(scores['test_recall_macro'])
        print(scores['test_accuracy'])
        validation_scores['KNeighbour'].append(scores['test_accuracy'])

    print(np.mean(training_scores['KNeighbour']))
    print(np.std(training_scores['KNeighbour']))
    # rest of print statments

我正在使用StratifiedKFold，但不确定是否需要像model_2函数中那样进行循环，或者在我们将cross_validate传递为参数时，cv=kfold函数是否已使用拆分。

我没有调用fit方法，可以吗？ cross_validate是自动呼叫还是在呼叫fit之前需要呼叫cross_validate？

最后，如何创建混淆矩阵？我是否需要为每个折叠创建它，如果是的话，如何计算最终/平均混淆矩阵？

Answer 1

model_1是正确的。

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html

cross_validate(estimator, X, y=None, groups=None, scoring=None, cv=’warn’, n_jobs=None, verbose=0, fit_params=None, pre_dispatch=‘2*n_jobs’, return_train_score=’warn’, return_estimator=False, error_score=’raise-deprecating’)

其中

estimator是实现“适合”的对象。它将被调用以使模型适合火车折叠。

cv：是一个交叉验证生成器，用于生成训练和测试拆分。

如果您遵循sklearn文档中的示例

cv_results = cross_validate(lasso, X, y, cv=3, return_train_score=False) cv_results['test_score'] array([0.33150734, 0.08022311, 0.03531764])

您可以看到模型lasso在火车拆分中每次折叠都适合3次，并且在测试拆分中也经过了3次验证。您可以看到报告了有关验证数据的测试分数。

Keras模型的交叉验证

Keras提供了包装器，使keras模型与sklearn cross_validatation方法兼容。您必须使用KerasClassifier

包装keras模型

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import KFold, cross_validate
from keras.models import Sequential
from keras.layers import Dense
import numpy as np

def get_model():
    model = Sequential()
    model.add(Dense(2, input_dim=2, activation='relu')) 
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

model = KerasClassifier(build_fn=get_model, epochs=10, batch_size=8, verbose=0)
kf = KFold(n_splits=3, shuffle=True)

X = np.random.rand(10,2)
y = np.random.rand(10,1)

cv_results = cross_validate(model, X, y, cv=kf, return_train_score=False)

print (cv_results)

Answer 2

在这些问题上，documentation可以说是您最好的朋友；从简单的示例中可以明显看出，您既不应使用for循环，也不应使用对fit的调用。修改示例以像实际一样使用KFold：

from sklearn.model_selection import KFold, cross_validate
from sklearn.datasets import load_boston
from sklearn.tree import DecisionTreeRegressor

X, y = load_boston(return_X_y=True)
n_splits = 5
kf = KFold(n_splits=n_splits, shuffle=True)

model = DecisionTreeRegressor()
scoring=('r2', 'neg_mean_squared_error')

cv_results = cross_validate(model, X, y, cv=kf, scoring=scoring, return_train_score=False)
cv_results

结果：

{'fit_time': array([0.00901461, 0.00563478, 0.00539804, 0.00529385, 0.00638533]),
 'score_time': array([0.00132656, 0.00214362, 0.00134897, 0.00134444, 0.00176597]),
 'test_neg_mean_squared_error': array([-11.15872549, -30.1549505 , -25.51841584, -16.39346535,
        -15.63425743]),
 'test_r2': array([0.7765484 , 0.68106786, 0.73327311, 0.83008371, 0.79572363])}

如何创建混淆矩阵？我是否需要为每折创建一个

没有人能告诉您是否需要为每个折叠创建混淆矩阵-这是您的选择。如果您选择这样做，最好跳过cross_validate并“手动”执行该过程-请在How to display confusion matrix and report (recall, precision, fmeasure) for each cross validation fold中查看我的答案。

如果是，如何计算最终/平均混淆矩阵？

没有“最终/平均”混淆矩阵；如果您想要计算除链接答案中所述的k以外的任何内容（每k折叠一个），则需要提供单独的验证集...

如何在scikit-learn中正确执行交叉验证？

2 个答案:

Keras模型的交叉验证