Question

例如，Keras＆＃39;的实施阿德格拉德一直是：

class Adagrad(Optimizer):
"""Adagrad optimizer.
It is recommended to leave the parameters of this optimizer
at their default values.
# Arguments
    lr: float >= 0. Learning rate.
    epsilon: float >= 0.
    decay: float >= 0. Learning rate decay over each update.
# References
    - [Adaptive Subgradient Methods for Online Learning and Stochastic Optimization](http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf)
"""

def __init__(self, lr=0.01, epsilon=1e-8, decay=0., **kwargs):
    super(Adagrad, self).__init__(**kwargs)
    self.lr = K.variable(lr)
    self.epsilon = epsilon
    self.decay = K.variable(decay)
    self.initial_decay = decay
    self.iterations = K.variable(0.)

def get_updates(self, params, constraints, loss):
    grads = self.get_gradients(loss, params)
    shapes = [K.get_variable_shape(p) for p in params]
    accumulators = [K.zeros(shape) for shape in shapes]
    self.weights = accumulators
    self.updates = []

    lr = self.lr
    if self.initial_decay > 0:
        lr *= (1. / (1. + self.decay * self.iterations))
        self.updates.append(K.update_add(self.iterations, 1))

    for p, g, a in zip(params, grads, accumulators):
        new_a = a + K.square(g)  # update accumulator
        self.updates.append(K.update(a, new_a))
        new_p = p - lr * g / (K.sqrt(new_a) + self.epsilon)
        # apply constraints
        if p in constraints:
            c = constraints[p]
            new_p = c(new_p)
        self.updates.append(K.update(p, new_p))
    return self.updates

功能＆＃39; get_update（）＆＃39;似乎是一步更新。但是，累加器是否应该存储历史信息？为什么它在每一步都被初始化为零？在整个培训过程中它如何成为累加器？

这条线做什么？

self.weights = accumulators

似乎self.weights再也没有被调用过。

Answer 1

你是对的..对于Keras中的所有优化器get_updates()实现了一步更新的张量逻辑。对model.fit() here中的每个_make_train_function()调用此函数一次，该函数用于通过将更新规则作为update= here传递来创建张量函数。此更新规则用于迭代迭代以更新模型参数和其他参数。

优化器类的

self.weights是其内部参数。这不用于培训。它只是用于保持优化器的状态（指向param / accumulators张量的指针列表）以及调用model.save时它们也通过调用get_weights() here来保存并被加载回来model.load here

调用set_weights()时

Adagrad如何在Keras徘徊？ Keras Optimizer中self.weights的含义是什么？

1 个答案: