我在Keras中有一个对象检测模型，希望基于验证集上计算出的平均平均精度（mAP）来监控我的训练。
我已将代码从tensorflow-models移植到脚本中，该脚本使用提供的模型和数据运行评估。不过，它不是作为Keras度量实现的，而是作为独立的类实现的：
struct Thing;
const SIZE: usize = 5;
fn main() {
let array: [Option<Box<Thing>>; SIZE] = [None, None, None, None, None];
}
拥有这样的东西我完全可以。确实，我不希望为训练批次计算它，因为它会减慢训练速度。
我的问题是：如何基于在每个时期后计算出的指标重复使用evaluation = SSDEvaluation(model, data, data_size)
mAP = evaluation.evaluate()
和ReduceLROnPlateau回调？
答案 0 :(得分：1)
您可以使用LambdaCallback来更新您的logs对象：
假设您的evaluation.evaluate()返回了像{'val/mAP': value}这样的字典，您可以这样做：
eval_callback = LambdaCallback(
on_epoch_end=lambda epoch, logs: logs.update(evaluation.evaluate())
)
这里的技巧是将logs进一步传递给其他回调，以便它们可以直接访问该值：
early_stopping = EarlyStopping(monitor='val/mAP', min_delta=0.0, patience=10, verbose=1, mode='max')
它将自动出现在CSVLogger和任何其他回调中。但是请注意，eval_callback必须使用回调列表中的值在任何回调之前：
callbacks = [eval_callback, early_stopping]
答案 1 :(得分：0)
我不确定SSDEvaluation是什么，但是如果可以接受没有开销的任何平均平均精度计算，我建议使用keras callbacks的以下方法。
您希望oto使用两个Callbacl的主要思想-EarlyStopping和ReduceLROnPlateau-都作用于纪元末尾并监视loss或metric的值。他们从method的logs参数中获得了这个值
def on_epoch_end(self, epoch, logs=None):
"""Called at the end of an epoch.
...
"""
-将实际的 map 发送到日志值，我们强制使用此方法以及从日志中获取准确性值的所有回调都将其使用。 Callbcaks从此处选择值（在代码中插入this-尽早停止，而this则是Reduce LR）。
因此，我们应该为两个回调“伪造”日志。我猜这不是理想的，但可行的解决方案。
此类从回调继承并计算 map 值，它们还避免了通过共享对象Hub重新计算 map 的情况。
from sklearn.metrics import average_precision_score
import keras
from keras.callbacks import Callback, EarlyStopping, ReduceLROnPlateau
class MAPHub:
def __init__(self):
self.map_value = None
-它只是共享 map 值的中心。可能会引起一些副作用。您可以尝试避免使用它。
def on_epoch_end(self, epoch, logs):
"""self just a callbcak instance"""
if self.last_metric_for_epoch == epoch:
map_ = self.hub.map_value
else:
prediction = self.model.predict(self._data, verbose=1)
map_ = average_precision_score(self._target, prediction)
self.hub.map_value = map_
self.last_metric_for_epoch = epoch
-此功能可以钙化并共享 map
class EarlyStoppingByMAP(EarlyStopping):
def __init__(self, data, target, hub, *args, **kwargs):
"""
data, target - values and target for the map calculation
hub - shared object to store _map_ value
*args, **kwargs for the super __init__
"""
# I've set monitor to 'acc' here, because you're interested in metric, not loss
super(EarlyStoppingByMAP, self).__init__(monitor='acc', *args, **kwargs)
self._target = target
self._data = data
self.last_metric_for_epoch = -1
self.hub = hub
def on_epoch_end(self, epoch, logs):
"""
epoch is the number of epoch, logs is a dict logs with 'loss' value
and metric 'acc' values
"""
on_epoch_end(self, epoch, logs)
logs['acc'] = self.hub.map_value # "fake" metric with calculated value
print('Go callback from the {}, logs: \n{}'.format(EarlyStoppingByMAP.__name__, logs))
super(EarlyStoppingByMAP, self).on_epoch_end(epoch, logs) # works as a callback fn
class ReduceLROnPlateauByMAP(ReduceLROnPlateau):
def __init__(self, data, target, hub, *args, **kwargs):
# the same as in previous
# I've set monitor to 'acc' here, because you're interested in metric, not loss
super(ReduceLROnPlateauByMAP, self).__init__(monitor='acc', *args, **kwargs)
self._target = target
self._data = data
self.last_metric_for_epoch = -1
self.hub = hub
def on_epoch_end(self, epoch, logs):
on_epoch_end(self, epoch, logs)
logs['acc'] = self.hub.map_value # "fake" metric with calculated value
print('Go callback from the {}, logs: \n{}'.format(ReduceLROnPlateau.__name__, logs))
super(ReduceLROnPlateauByMAP, self).on_epoch_end(epoch, logs) # works as a callback fn
- NB 不要在构造函数中使用monitor参数！您应该使用'acc'，参数已经设置为正确的值。
一些测试：
from keras.datasets import mnist
from keras.models import Model
from keras.layers import Dense, Input
import numpy as np
(X_tr, y_tr), (X_te, y_te) = mnist.load_data()
X_tr = (X_tr / 255.).reshape((60000, 784))
X_te = (X_te / 255.).reshape((10000, 784))
def binarize_labels(y):
y_bin = np.zeros((len(y), len(np.unique(y))))
y_bin[range(len(y)), y] = 1
return y_bin
y_train_bin, y_test_bin = binarize_labels(y_tr), binarize_labels(y_te)
inp = Input(shape=(784,))
x = Dense(784, activation='relu')(inp)
x = Dense(256, activation='relu')(x)
out = Dense(10, activation='softmax')(x)
model = Model(inp, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
-一个简单的“测试套件”。现在适合它：
hub = MAPHub() # instentiate a hub
# I will use default params except patience as example, set it to 1 and 5
early_stop = EarlyStoppingByMAP(X_te, y_test_bin, hub, patience=1) # Patience is EarlyStopping's param
reduce_lt = ReduceLROnPlateauByMAP(X_te, y_test_bin, hub, patience=5) # Patience is ReduceLR's param
history = model.fit(X_tr, y_train_bin, epochs=10, callbacks=[early_stop, reduce_lt])
Out:
Epoch 1/10
60000/60000 [==============================] - 12s 207us/step - loss: 0.1815
10000/10000 [==============================] - 1s 59us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.18147853660446903, 'acc': 0.9934216252519924}
Epoch 2/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0784
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.07844233275586739, 'acc': 0.9962269038764738}
Epoch 3/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0556
10000/10000 [==============================] - 0s 40us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
10000/10000 [==============================] - 0s 40us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.05562876497630107, 'acc': 0.9972085346550085}
Epoch 4/10
60000/60000 [==============================] - 12s 198us/step - loss: 0.0389
10000/10000 [==============================] - 0s 41us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
10000/10000 [==============================] - 0s 41us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.0388911374788188, 'acc': 0.9972696414934574}
Epoch 5/10
60000/60000 [==============================] - 12s 197us/step - loss: 0.0330
10000/10000 [==============================] - 0s 39us/step
Go callback from the EarlyStoppingByMAP, logs:
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}
10000/10000 [==============================] - 0s 39us/step
Go callback from the ReduceLROnPlateau, logs:
{'loss': 0.03298293751536124, 'acc': 0.9959456176387349}
好吧，至少看起来像是在尽早停止。我猜是ReduceLROnPlateau的原因，因为它们使用相同的日志和相似的逻辑-如果设置了适当的参数。
如果您不想使用sklearn函数，而是使用SSDEvaluation（我只是找不到它），那么您可以轻松地使用on_epoch_method函数来处理此评估函数。
希望有帮助。