Question

我想实现循环学习率，而不是AdamOptimizer或任何其他形式的SGD。因此，我想在这里介绍的部分是＆＃34;循环学习率＆＃34;，功能get_triangular_lr。功能如下所示：

def get_triangular_lr(iteration, stepsize, base_lr, max_lr):
    """Given the inputs, calculates the lr that should be applicable for 
       this iteration"""
    scale_fn = lambda x: 1/(2.**(x-1))
    cycle = math.floor(1 + iteration/(2  * stepsize))
    x = abs(iteration/stepsize - 2 * cycle + 1)
    lr = base_lr + (max_lr - base_lr) * max(0, (1-x)) *scale_fn(cycle)
    lr = tf.convert_to_tensor(lr)

    return lr

下面，显示了在实际的Inception ResNet V2模型中使用它的代码，我的目的是用循环LR替换tf_train_exponential_decay，循环LR本身会增加和减少每步采用的学习速率。

      iteration_step = 0 #(is increased every global_step)
      lr1 = get_triangular_lr(iteration_step, stepsize, base_lr, max_lr)


    #Define your exponentially decaying learning rate
#         lr = tf.train.exponential_decay(
#             learning_rate = initial_learning_rate,
#             global_step = global_step,
#             decay_steps = decay_steps,
#             decay_rate = learning_rate_decay_factor,
#             staircase = True)

    #Now we can define the optimizer that takes on the learning rate

    optimizer = tf.train.AdamOptimizer(learning_rate = lr1)

    #Create the train_op.
    train_op = slim.learning.create_train_op(total_loss, optimizer)

但是，当我运行上面的代码时（通过运行完整的训练模型），使用该功能不会增加学习速度。它只从0.0001开始，并且每个global_step都使用此值，而它应该增加每个全局步骤（使用get_triangular_lr计算的新值。我认为它需要某种for循环，但我无法弄清楚如何实现它。

    #Now we need to create a training step function that runs both the 
    #train_op, metrics_op and updates the global_step concurrently.

    def train_step(sess, train_op, global_step):
        '''
        Simply runs a session for the three arguments provided and gives a 
     logging on the time elapsed for each global step
        '''
        #Check the time for each sess run
        start_time = time.time()
        total_loss, global_step_count, _ = sess.run([train_op, global_step, 
        metrics_op])
        time_elapsed = time.time() - start_time

        #Run the logging to print some results
        logging.info('global step %s: loss: %.4f (%.2f sec/step)', 
        global_step_count, total_loss, time_elapsed)

        return total_loss, global_step_count

当我将train_op的输入更改为total_loss, lr1时，它会给我一个AttributeError：＆＃39; Tensor＆＃39;对象没有属性＆＃39; compute_gradients＆＃39;，因此模型根本不运行。

编辑：尚未找到解决方案，但设法使用py.func，但现在出现下一个错误：

所以我创建了triangular_tf，它使用tf.py_func来加载函数，如下所示：

triangular_tf = tf.py_func(get_triangular_lr, [iteration, stepsize, base_lr, max_lr], [tf.float64])

唯一的问题是，当我现在将triangular_tf传递给`optimizer = tf.train.AdamOptimizer（learning_rate = triangular_tf）时，会发生以下错误：

ValueError: cannot add op with name <my weights variable name>/Adam as that name is already used

我不确定在这个过程中是否还需要AdamOptimizer，但我只是需要一种方法来调整学习速度。

实施变量学习率Tensorflow

0 个答案: