Question

我试图首先单独训练一个语言模型（类PTBModel），然后附加一个分类器，将语言模型偏向情感分类，就像它们在http://deeplearning.net/tutorial/lstm.html中那样。当所有变量都以相同的学习率训练时，我能够做到这一点，但我希望语言模型的学习率较低，因为它已经过训练。我在另一篇文章中找到了解决方案，但是当我将变量分开如下时：

# Get trainable variables for the language model
with tf.variable_scope("RNN") as vs:
    lm_parameters = [v for v in tf.trainable_variables()
            if v.name.startswith(vs.name)]

with tf.variable_scope("RNN_SOFTMAX") as vs:
   lm_parameters = [v for v in tf.trainable_variables()
           if v.name.startswith(vs.name)]



# Get trainable variables for the classifier
with tf.variable_scope("output_projection") as vs:
    classifier_parameters = [v for v in tf.trainable_variables()
            if v.name.startswith(vs.name)]

然后计算并尝试应用渐变。

gradients = tf.gradients(self.losses, lm_parameters + classifier_parameters)
clipped_gradients, norm = tf.clip_by_global_norm(gradients, self.max_gradient_norm)

with tf.name_scope("grad_norms") as scope:
     grad_summ = tf.scalar_summary("grad_norms", norm)

print(gradients)
print(clipped_gradients)

grads1 = clipped_gradients[:len(lm_parameters)]
grads2 = clipped_gradients[len(lm_parameters):]

train_op1 = opt_lm.apply_gradients(zip(grads1, lm_parameters))
train_op2 = opt_classifier.apply_gradients(zip(grads2, classifier_parameters))

self.update = train_op2

我收到以下错误

[None, None, <tensorflow.python.framework.ops.Tensor object at 0x7fb6cded3f50>, <tensorflow.python.framework.ops.Tensor object at 0x7fb6cded3790>]
[None, None, <tensorflow.python.framework.ops.Tensor object at 0x7fb6cde92190>, <tensorflow.python.framework.ops.Tensor object at 0x7fb6cde92490>]
Traceback (most recent call last):
  File "train.py", line 214, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "train.py", line 210, in main
    train_sentiment()
  File "train.py", line 125, in train_sentiment
    , is_training=True, config=config)
  File "/home/seberik/workspace/lm-generator/models/sentiment.py", line 159, in __init__
    train_op1 = opt_lm.apply_gradients(zip(grads1, lm_parameters))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 277, in apply_gradients
    (grads_and_vars,))
ValueError: No gradients provided for any variable: ((None, <tensorflow.python.ops.variables.Variable object at 0x7fb6e0b14250>), (None, <tensorflow.python.ops.variables.Variable object at 0x7fb6e0b143d0>))

我不知道为什么会发生这种情况，因为以下代码可以正常工作

gradients               = tf.gradients(self.losses, params)
clipped_gradients, norm = tf.clip_by_global_norm(gradients, self.max_gradient_norm)
with tf.name_scope("grad_norms") as scope:
    grad_summ = tf.scalar_summary("grad_norms", norm)
self.update = opt.apply_gradients(zip(clipped_gradients, params), global_step=self.global_step)
loss_summ = tf.scalar_summary("{0}_loss".format(self.str_summary_type), self.mean_loss)
acc_summ = tf.scalar_summary("{0}_accuracy".format(self.str_summary_type), self.accuracy)
self.merged = tf.merge_summary([loss_summ, acc_summ])

整个代码位于以下存储库中，非常感谢有关此的一些输入。 https://bitbucket.org/briq/lm-generator/src

不同的学习率会导致错误：没有为任何变量提供渐变

0 个答案: