Question

如何在Tensorflow 2.0 / Keras中（平整的）conv2d层之后添加LSTM层？我的训练输入数据具有以下形状（大小，sequence_length，高度，宽度，通道）。对于卷积层，我一次只能处理一个图像，对于LSTM层，我需要一系列功能。有没有一种方法可以在LSTM层之前重塑数据，以便您可以将两者结合在一起？

Answer 1

从您提供的形状概述(size, sequence_length, height, width, channels)看来，每个标签都有图像序列。为此，我们通常使用Conv3D。我在下面附上示例代码：

import tensorflow as tf

SIZE = 64
SEQUENCE_LENGTH = 50
HEIGHT = 128
WIDTH = 128
CHANNELS = 3

data = tf.random.normal((SIZE, SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))

input = tf.keras.layers.Input((SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
hidden = tf.keras.layers.Conv3D(32, (3, 3, 3))(input)
hidden = tf.keras.layers.Reshape((-1, 32))(hidden)
hidden = tf.keras.layers.LSTM(200)(hidden)

model = tf.keras.models.Model(inputs=input, outputs=hidden)
model.summary()

输出：

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 50, 128, 128, 3)] 0         
_________________________________________________________________
conv3d (Conv3D)              (None, 48, 126, 126, 32)  2624      
_________________________________________________________________
reshape (Reshape)            (None, None, 32)          0         
_________________________________________________________________
lstm (LSTM)                  (None, 200)               186400    
=================================================================
Total params: 189,024
Trainable params: 189,024
Non-trainable params: 0

如果您仍然想使用Conv2D（在您的情况下不建议使用），则必须执行如下所示的操作。基本上，您将在高度维度上附加图像序列，这将使您失去时间维度。

import tensorflow as tf

SIZE = 64
SEQUENCE_LENGTH = 50
HEIGHT = 128
WIDTH = 128
CHANNELS = 3

data = tf.random.normal((SIZE, SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))

input = tf.keras.layers.Input((SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
hidden = tf.keras.layers.Reshape((SEQUENCE_LENGTH * HEIGHT, WIDTH, CHANNELS))(input)
hidden = tf.keras.layers.Conv2D(32, (3, 3))(hidden)
hidden = tf.keras.layers.Reshape((-1, 32))(hidden)
hidden = tf.keras.layers.LSTM(200)(hidden)

model = tf.keras.models.Model(inputs=input, outputs=hidden)
model.summary()

输出：

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 50, 128, 128, 3)] 0         
_________________________________________________________________
reshape (Reshape)            (None, 6400, 128, 3)      0         
_________________________________________________________________
conv2d (Conv2D)              (None, 6398, 126, 32)     896       
_________________________________________________________________
reshape_1 (Reshape)          (None, None, 32)          0         
_________________________________________________________________
lstm (LSTM)                  (None, 200)               186400    
=================================================================
Total params: 187,296
Trainable params: 187,296
Non-trainable params: 0
_________________________________________________________________

Tensorflow 2.0结合CNN + LSTM

1 个答案: