Question

我使用Keras（Python）作为CNN模型，并具有自定义回调函数来计算精度，召回等指标。我已经给出了下面的功能。此函数适用于fit函数的单个调用，并在epoch结束时返回正确的度量值。但是，我正在尝试使用GridSearchCV函数进行K-Fold交叉验证，并希望为每个折叠返回自定义度量标准，并且当我使用grid_result.grid_scores_来获取K-Folds的均值/标准时。这不起作用。希望得到一些帮助。

此外，是否可以获取针对单独验证数据计算的指标。

from keras.callbacks import Callback
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score

class CustomMetrics(Callback):

def on_train_begin(self, logs={}):
    self.validation_f1 = []
    self.validation_recall = []
    self.validation_precision = []

def on_epoch_end(self, epoch, logs={}):
    # Extracting the second column since the activation is softmax
    predicted_prob = self.model.predict(self.validation_data[0])[:,1]
    true_y = self.validation_data[1][:,1]

    threshold = optimal_cutoff_PRC(true_y,predicted_prob)
    convert_to_01 = lambda x: 1 if x > threshold else 0
    vfunc = numpy.vectorize(convert_to_01)
    test_predicted_01 = vfunc(predicted_prob)         

    _val_f1 = f1_score(true_y, test_predicted_01)
    _val_recall = recall_score(true_y, test_predicted_01)
    _val_precision = precision_score(true_y, test_predicted_01)

    self.validation_f1.append(_val_f1)
    self.validation_recall.append(_val_recall)
    self.validation_precision.append(_val_precision)
    print " — val_f1: %f — val_precision: %f — val_recall %f" %(_val_f1, _val_precision, _val_recall)
    return

    custom_metrics = CustomMetrics()

以下调用适用于上述函数：

model.fit(timeseries_train, y_train, batch_size = 512, epochs=2, callbacks = [custom_metrics]   , validation_data= val_data)

但是，对于GridSearchCV函数定义如下，这不能正常工作：

seed = 7
numpy.random.seed(seed)
model = KerasClassifier(build_fn=create_cnn_model,batch_size = 512, epochs=1, verbose=1)
learning_rate = [0.001]
cnn_filters = [64]
cnn_kernel_size = [5]
param_grid = dict(learning_rate=learning_rate,cnn_filters=cnn_filters,cnn_kernel_size=cnn_kernel_size)
grid = GridSearchCV(estimator=model, param_grid=param_grid, fit_params =    dict(callbacks=[ custom_metrics]),n_jobs=1)
grid_result = grid.fit(timeseries_train, y_train,validation_data = val_data)
param_grid

grid_result.grid_scores_ only

以上命令会返回常规指标的准确性，并且不会返回任何自定义指标：

[mean: 0.99428, std: 0.00019, params: {'cnn_kernel_size': 5, 'learning_rate': 0.001, 'cnn_filters': 64}]

它既不返回K-fold的自定义指标，也不返回传递的其他验证数据。

网格在Keras中使用自定义指标进行搜索

0 个答案: