Question

我正在线上学习基于动量的学习教程，并在Theano中找到了这种方法

def gradient_updates_momentum(cost, params, learning_rate, momentum):
    '''
Compute updates for gradient descent with momentum

:parameters:
    - cost : theano.tensor.var.TensorVariable
        Theano cost function to minimize
    - params : list of theano.tensor.var.TensorVariable
        Parameters to compute gradient against
    - learning_rate : float
        Gradient descent learning rate
    - momentum : float
        Momentum parameter, should be at least 0 (standard gradient descent) and less than 1

:returns:
    updates : list
        List of updates, one for each parameter
'''
# Make sure momentum is a sane value
assert momentum < 1 and momentum >= 0
# List of update steps for each parameter
updates = []
# Just gradient descent on cost
for param in params:
    # For each parameter, we'll create a param_update shared variable.
    # This variable will keep track of the parameter's update step across iterations.
    # We initialize it to 0
    param_update = theano.shared(param.get_value()*0., broadcastable=param.broadcastable)
    # Each parameter is updated by taking a step in the direction of the gradient.
    # However, we also "mix in" the previous step according to the given momentum value.
    # Note that when updating param_update, we are using its old value and also the new gradient step.
    updates.append((param, param - learning_rate*param_update))
    # Note that we don't need to derive backpropagation to compute updates - just use T.grad!
    updates.append((param_update, momentum*param_update + (1. - momentum)*T.grad(cost, param)))
return updates

以下两行的顺序不应该是相反的方式（互换）吗？

updates.append((param, param - learning_rate*param_update))

和

updates.append((param_update, momentum*param_update + (1. - momentum)*T.grad(cost, param)))

据我所知，在执行列车方法并计算成本后，才会运行更新，对吗？

这是不是意味着我们应该使用当前成本，并且使用现有的param_update值（来自前一次迭代），我们应该计算更新的param_update，从而更新当前的param值？

为什么反之亦然？为什么这是正确的？

Answer 1

忽略提供给theano.function的更新列表中的更新顺序。始终使用共享变量的旧值计算更新。

此代码段显示忽略更新顺序：

import theano
import theano.tensor

p = 0.5
param = theano.shared(1.)
param_update = theano.shared(2.)
cost = 3 * param * param
update_a = (param, param - param_update)
update_b = (param_update, p * param_update + (1 - p) * theano.grad(cost, param))
updates1 = [update_a, update_b]
updates2 = [update_b, update_a]
f1 = theano.function([], outputs=[param, param_update], updates=updates1)
f2 = theano.function([], outputs=[param, param_update], updates=updates2)
print f1(), f1()
param.set_value(1)
param_update.set_value(2)
print f2(), f2()

从逻辑上讲，如果你想要

new_a = old_a + a_update
new_b = new_a + b_update

然后你需要提供这样的更新：

new_a = old_a + a_update
new_b = old_a + a_update + b_update

在以下Theano方法中更新参数的方式是否有错误？

1 个答案: