Question

假设训练已完成：推断时（在LSTM和RNN层中）Keras对第0个单元状态和隐藏状态使用什么值？我可以想到至少三种情况，并且在文档中找不到任何结论性的答案：

（a）学习初始状态，然后将其用于所有预测

（b）或初始状态始终设置为零

（c）初始状态始终是随机的（希望不是...）

Answer 1

如果使用LSTM(stateful=True)，则隐藏状态将初始化为零，用fit或predict进行更改，并保持不变，直到调用.reset_states()为止。如果为LSTM(stateful=False)，则在对每个批次进行拟合/预测/等后重置状态。

可以从.reset_states() source code进行验证，并可以直接检查；两者均适用于以下stateful=True。有关如何传递状态的更多信息，请参见this answer。

直接检查：

batch_shape = (2, 10, 4)
model = make_model(batch_shape)

X = np.random.randn(*batch_shape)
y = np.random.randint(0, 2, (batch_shape[0], 1))

show_lstm_states("STATES INITIALIZED")
model.train_on_batch(X, y)

show_lstm_states("STATES AFTER TRAIN")
model.reset_states()
show_lstm_states("STATES AFTER RESET")

model.predict(X)
show_lstm_states("STATES AFTER PREDICT")

输出：

STATES INITIALIZED
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]]
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]]

STATES AFTER TRAIN
[[0.12061571 0.03639204 0.20810013 0.05309075]
 [0.01832913 0.00062357 0.10566339 0.60108346]]
[[0.21241754 0.0773523  0.37392718 0.15590034]
 [0.08496398 0.00112716 0.23814857 0.95995367]]

STATES AFTER RESET
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]]
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]]

STATES AFTER PREDICT
[[0.12162527 0.03720453 0.20628096 0.05421837]
 [0.01849432 0.00064993 0.1045063  0.6097021 ]]
[[0.21398112 0.07894284 0.3709934  0.15928769]
 [0.08605779 0.00117485 0.23606434 0.97212094]]

使用的功能/导入：

import tensorflow as tf
import tensorflow.keras.backend as K
from tensorflow.keras.layers import Input, Dense, LSTM
from tensorflow.keras.models import Model
import numpy as np

def make_model(batch_shape):
    ipt = Input(batch_shape=batch_shape)
    x   = LSTM(4, stateful=True, activation='relu')(ipt)
    out = Dense(1, activation='sigmoid')(x)

    model = Model(ipt, out)
    model.compile('adam', 'binary_crossentropy')

    return model

def show_lstm_states(txt=''):
    print('\n' + txt) 
    states = model.layers[1].states

    for state in states:
        if tf.__version__[0] == '2':
            print(state.numpy())
        else:
            print(K.get_value(state))

检查源代码：

from inspect import getsource
print(getsource(model.layers[1].reset_states))

Answer 2

我对this的理解是，大多数情况下它们都被初始化为零。

Keras对单元状态的初始值（LSTM）和初始隐藏状态（RNN，LSTM）进行了什么处理？

2 个答案: