Question

所有

我在seq2seq任务中使用了类似于bucketing的技术：

# For different length in encoder and decoder
model_map = {}
for i in encoder_shape:
    for j in decoder_shape:
        with variable_scope.variable_scope(variable_scope.get_variable_scope(),
                                 reuse=True if tt > 0 else None):
            model = Seq2SeqModel()
            model.build(encoder[:i], decoder[:j])
            model_map[i*100+j] = model

获取共享模型的参数：

for t in tf.all_variables():
    print t.name, t.get_shape() 

Print: 
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding:0 (50000, 256)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias:0 (1600,)

模型的优化器如下所示：

#every model have an optimizer
params = tf.trainable_variables()
opt = tf.train.AdamOptimizer(1e-3)
gradients = tf.gradients(self.loss, params)
self.optimizer = opt.apply_gradients(zip(gradients, params))

但我发现优化器不共享变量：

embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam:0 (50000, 256)
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam_1:0 (50000, 256)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam_1:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam:0 (1600,)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam_1:0 (1600,)
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam_2:0 (50000, 256)
embedding_attention_seq2seq/RNN/EmbeddingWrapper/embedding/Adam_3:0 (50000, 256)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam_2:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Matrix/Adam_3:0 (1056, 1600)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam_2:0 (1600,)
embedding_attention_seq2seq/RNN/MultiRNNCell/Cell0/GRUCell/Gates/Linear/Bias/Adam_3:0 (1600,)

随着桶数量的增长，GPU内存也将增长。同时我在tf.train.Saver.save（）中得到了一个更大的模型。

那么可以在张量流中分享梯度吗？

Answer 1

我相信在所有模型中共享一个优化器实例可以完成您想要的任务。

在Tensorflow中使用bucketing时，如何在Adam优化器中共享渐变和变量？

1 个答案: