Keras预测给出的误差与评估不同,损失与指标不同

时间:2018-04-18 15:35:01

标签: validation tensorflow keras loss metric

我有以下问题:

我在Keras有一个自动编码器,并训练了几个时代。培训概述显示验证MAE为0.0422,MSE为0.0024。 但是,如果我然后调用network.predict并手动计算验证错误,我得到0.035和0.0024。

人们会认为我对MAE的手动计算是不正确的,但奇怪的是,如果我使用身份模型(只输出您输入的内容)并使用它来评估预测值,则相同的误差值是返回我的手动计算。代码如下:

input = Input(shape=(X_train.shape[1], ))
encoded = Dense(50, activation='relu', activity_regularizer=regularizers.l1(10e-5))(input)
encoded = Dense(50, activation='relu', activity_regularizer=regularizers.l1(10e-5))(encoded)
encoded = Dense(50, activation='relu', activity_regularizer=regularizers.l1(10e-5))(encoded)
decoded = Dense(50, activation='relu', activity_regularizer=regularizers.l1(10e-5))(encoded)
decoded = Dense(50, activation='relu', activity_regularizer=regularizers.l1(10e-5))(decoded)
decoded = Dense(X_train.shape[1], activation='sigmoid')(decoded)
network = Model(input, decoded)

# sgd = SGD(lr=8, decay=1e-6)
# network.compile(loss='mean_squared_error', optimizer='adam')
network.compile(loss='mean_absolute_error', optimizer='adam', metrics=['mse'])

# Fitting the data
network.fit(X_train, X_train, epochs=2, batch_size=1, shuffle=True, validation_data=(X_valid, X_valid),
            callbacks=[EarlyStopping(monitor='val_loss', min_delta=0.00001, patience=20, verbose=0, mode='auto')])


# Results
recon_valid = network.predict(X_valid, batch_size=1)
score2 = network.evaluate(X_valid, X_valid, batch_size=1, verbose=0)
print('Network evaluate result: mae={}, mse={}'.format(*score2))

x = Input((X_train.shape[1],))
m = Model(x, x)
m.compile(loss='mean_absolute_error', optimizer='adam', metrics=['mse'])
score1 = m.evaluate(recon_valid, X_valid, batch_size=1, verbose=0)
print('Identity evaluate result: mae={}, mse={}'.format(*score1))

errors_test = np.absolute(X_valid - recon_valid)
print("Manual MAE: {}".format(np.average(errors_test)))
errors_test = np.square(X_valid - recon_valid)
print("Manual MSE: {}".format(np.average(errors_test)))

其中输出以下内容:

Train on 282 samples, validate on 94 samples
Epoch 1/2
2018-04-18 17:24:01.464947: I C:\tf_jenkins\workspace\rel-win\M\windows\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
282/282 [==============================] - 0s - loss: 0.0861 - mean_squared_error: 0.0187 - val_loss: 0.0451 - val_mean_squared_error: 0.0025
Epoch 2/2
282/282 [==============================] - 0s - loss: 0.0440 - mean_squared_error: 0.0025 - val_loss: 0.0422 - val_mean_squared_error: 0.0024
Network evaluate result: mae=0.04216482736011769, mse=0.0024067993242382767
Identity evaluate result: mae=0.03506102238563781, mse=0.0024067993242382767
Manual MAE: 0.03506102412939072
Manual MSE: 0.002406799467280507

我知道我的手动计算是正确的,因为身份模型(m)返回相同的值。如果network.evaluate(X_valid,X_valid)以某种方式使用与network.predict(X_valid)返回的值不同的值,那么MAE值差异的唯一可能解释是,但是MSE也将是不同的。

这让我完全感到困惑,认为Keras MAE计算中可能存在错误。有没有人之前有这个问题或有任何想法如何解决?我正在使用Tensorflow后端。 任何帮助将不胜感激!

编辑:我几乎可以肯定这是一个错误。如果我保持loss ='mae'但也添加metrics = ['mse','mae'],则指标返回的MAE与我的手动计算和身份模型相同。对于MSE也是如此:如果我设置了loss ='mse',则度量返回的MSE与损失不同。

2 个答案:

答案 0 :(得分:2)

事实证明,由于正规化,损失应该与度量不同。使用正则化,损失更高(在我的情况下),因为当节点不像指定的那样活跃时,正则化会增加损失。指标不考虑这一点,因此会返回一个不同的值,该值等于手动计算错误时的值。

答案 1 :(得分:0)

The metrics during training and validation are different because of different reasons:

  • The dataset is different
  • During trainning the weights are changing in every step so the metrics are changing too
  • The metric during training is of the current batch of data or a running average of the metrics of the last batches. For the evaluation, the metric is for the whole dataset.