将DropoutWrapper和ResidualWrapper与variational_recurrent = True相结合

时间:2018-04-22 20:18:53

标签: tensorflow lstm recurrent-neural-network dropout deep-residual-networks

我正在尝试创建一个用DropoutWrapper和ResidualWrapper包装的LSTM单元格的MultiRNNCell。对于使用variational_recurrent = True,我们必须向DropoutWrapper提供input_size参数。我无法弄清楚应该将input_size传递给每个LSTM层,因为ResidualWrapper还添加了跳过连接以增加每层的输入。

我正在使用以下实用程序功能来创建一个LSTM图层:

def create_cell(units, residual_connections, keep_prob, input_size):
    lstm_cell = tf.nn.rnn_cell.LSTMCell(units,
                                        activation=tf.nn.tanh,
                                        initializer=tf.truncated_normal_initializer(),
                                        cell_clip=5.)
    lstm_cell = tf.nn.rnn_cell.DropoutWrapper(lstm_cell,
                                              dtype=tf.float32,
                                              input_keep_prob=keep_prob,
                                              output_keep_prob=keep_prob,
                                              state_keep_prob=keep_prob,
                                              variational_recurrent=True,
                                              input_size=input_size)

    if residual_connections:
        lstm_cell = tf.nn.rnn_cell.ResidualWrapper(lstm_cell)
    return lstm_cell

以下代码创建完整的单元格:

net = tf.layers.dense(inputs,
                      128,
                      activation=tf.nn.relu, 
                      kernel_initializer=tf.variance_scaling_initializer())
net = tf.layers.batch_normalization(net, training=training)    
cells = [create_cell(64, False, keep_prob, ??)]
for _ in range(5):
    cells.append(create_cell(64, True, keep_prob, ??))
multirnn_cell = tf.nn.rnn_cell.MultiRNNCell(cells)
net, rnn_s1 = tf.nn.dynamic_rnn(cell=multirnn_cell, inputs=net, initial_state=rnn_s0, dtype=tf.float32)

对于第一个和后续LSTM图层,应该将哪些值传递给input_size?

0 个答案:

没有答案