Keras TimeDistributed用于多输入情况？

Question

模型说明

在我们的模型中，我想时间将low_level_model分配到LSTM上层以创建分层模型。 low_level_model通过汇总区域序列的结果及其visit_id来找到客户访问的隐藏表示。每个区域序列都经过CNN和关注层，并将结果与​​每次访问的嵌入向量相连接。

据我所知，TimeDistributed包装器可用于构建分层模型，因此我尝试用两个不同的输入包装我们的low_level_model。但似乎该库不支持多输入案例。这是我们的代码。

# Get 1st input
visit_input = keras.Input((1,))
visit_emb = visit_embedding_layer(visit_input)
visit_output = Reshape((-1,))(visit_emb)

# Get 2nd input - Shallow model
areas_input = keras.Input((10,))
areas_emb = area_embedding_layer(areas_input)
areas_cnn = Conv1D(filters=200, kernel_size=5,
               padding='same', activation='relu', strides=1)(areas_emb)
areas_output = simple_attention(areas_cnn, areas_cnn)

# Concat two results from 1st and 2nd input
v_a_emb_concat = Concatenate()([visit_output, areas_output])

# Define this model as low_level_model
low_level_model = keras.Model(inputs=[areas_input, visit_input], outputs=v_a_emb_concat)

# Would like to use the result of this low_level_model as inputs for higher-level LSTM layer.
# Therefore, wrap this model by TimeDistributed layer
encoder = TimeDistributed(low_level_model)

# New input with step-size 5 (Consider 5 as the number of previous data)
all_visit_input = keras.Input((5, 1))
all_areas_input = keras.Input((5, 10))

# This part raises AssertionError (assert len(input_shape) >= 3)
all_areas_rslt = encoder(inputs=[all_visit_input, all_areas_input])
all_areas_lstm = LSTM(64, return_sequences=False)(all_areas_rslt)
logits = Dense(365, activation='softmax')(all_areas_lstm)

# Model define (Multi-input ISSUE HERE!)
self.model = keras.Model(inputs=[all_visit_input, all_areas_input], outputs=logits)

self.model.compile(optimizer=keras.optimizers.Adam(0.001),
                   loss=custom_loss_function)

# Get data
self.train_data = data.train_data_generator_hist()
self.test_data = data.test_data_generator_hist()

# Fit
self.history = self.model.fit_generator(
generator=self.train_data,
steps_per_epoch=train_data_size//FLAGS.batch_size,
epochs=FLAGS.train_epochs]
)

错误消息

错误消息如下。

File "/home/dmlab/sundong/revisit/survival-revisit-code/survrev.py", line 163, in train_test
all_areas_rslt = encoder(inputs=[all_visit_input, all_areas_input])
File "/home/dmlab/ksedm1/anaconda3/envs/py36/lib/python3.6/site-packages/keras/engine/base_layer.py", line 431, in __call__
self.build(unpack_singleton(input_shapes))
File "/home/dmlab/ksedm1/anaconda3/envs/py36/lib/python3.6/site-packages/keras/layers/wrappers.py", line 195, in build
assert len(input_shape) >= 3
AssertionError

我尝试过的

1）我读了此keras issue，但不清楚如何提出转发多个输入的技巧。

2）我只使用单个输入（例如，TimeDistribute）时检查了areas_input的代码是否有效。修改后的代码示例如下。

3）现在尝试遵循[上一个问题]。 （Keras TimeDistributed layer with multiple inputs）

# Using only one input 
areas_input = keras.Input((10,))
areas_emb = area_embedding_layer(areas_input)
areas_cnn = Conv1D(filters=200, kernel_size=5,
           padding='same', activation='relu', strides=1)(areas_emb)
areas_output = simple_attention(areas_cnn, areas_cnn)

# Define this model as low_level_model
low_level_model = keras.Model(inputs=areas_input, outputs=areas_output)

# Would like to use the result of this low_level_model as inputs for higher-level LSTM layer.
# Therefore, wrap this model by TimeDistributed layer
encoder = TimeDistributed(low_level_model)

# New input with step-size 5 (Consider 5 as the number of previous data)
all_areas_input = keras.Input((5, 10))

# No Error
all_areas_rslt = encoder(inputs=all_areas_input)
all_areas_lstm = LSTM(64, return_sequences=False)(all_areas_rslt)
logits = Dense(365, activation='softmax')(all_areas_lstm)

# Model define (Multi-input ISSUE HERE!)
self.model = keras.Model(inputs=all_areas_input, outputs=logits)

self.model.compile(optimizer=keras.optimizers.Adam(0.001),
               loss=custom_loss_function)

# Get data
self.train_data = data.train_data_generator_hist()
self.test_data = data.test_data_generator_hist()

# Fit
self.history = self.model.fit_generator(
generator=self.train_data,
steps_per_epoch=train_data_size//FLAGS.batch_size,
epochs=FLAGS.train_epochs]
)

提前感谢您分享解决此问题的技术。

Answer 1

总而言之，我通过完全获取输入来解决此问题，并使用Lambda层划分了这些输入。 TimeDistributed只能接受单个输入，这就是为什么。这是我的代码段。

single_input = keras.Input((1+10),))
visit_input = Lambda(lambda x: x[:, 0:1])(single_input)
areas_input = Lambda(lambda x: x[:, 1: ])(single_input)
...
low_level_model = keras.Model(inputs=single_input, outputs=concat)

encoder = TimeDistributed(low_level_model)
multiple_inputs = keras.Input((5, 11)))
all_areas_rslt = encoder(inputs=multiple_inputs)
all_areas_lstm = LSTM(64, return_sequences=False)(all_areas_rslt)
logits = Dense(365, activation='softmax')(all_areas_lstm)

Answer 2

我得到了相同的错误消息，并将其追溯到与原始海报相同的问题和Github问题。通过使用TimeDistributed，我能够用keras.layers.RepeatVector输出层解决多个输入的问题。下面是我的示例：

core_input_1 = Input(shape=(self.core_timesteps, self.core_input_1_dim), name='core_input_1')
core_branch_1 = BatchNormalization(momentum=0.0, name='core_1_bn')(core_input_1)
core_branch_1 = LSTM(self.core_nodes[0], activation='relu', name='core_1_lstm_1', return_sequences=True)(core_branch_1)
core_branch_1 = LSTM(self.core_nodes[1], activation='relu', name='core_1_lstm_2')(core_branch_1)

core_input_2 = Input(shape=(self.core_timesteps, self.core_input_2_dim), name='core_input_2')
core_branch_2 = BatchNormalization(momentum=0.0, name='core_2_bn')(core_input_2)
core_branch_2 = LSTM(self.core_nodes[0], activation='relu', name='core_2_lstm_1', return_sequences=True)(core_branch_2)
core_branch_2 = LSTM(self.core_nodes[1], activation='relu', name='core_2_lstm_2')(core_branch_2)

merged = Concatenate()([core_branch_1, core_branch_2])

full_branch = RepeatVector(self.output_timesteps)(merged)        
full_branch = LSTM(self.core_nodes[1], activation='relu', name='final_lstm', return_sequences=True)(full_branch)

full_branch = TimeDistributed(Dense(self.output_dim, name='td_dense', activation='relu'))(full_branch)
full_branch = TimeDistributed(Dense(self.output_dim, name='td_dense'))(full_branch)

model = Model(inputs=[core_input_1, core_input_2], outputs=full_branch, name='full_model')

我张贴了完整的示例，以便其他人可以看到，但是解决方案的关键部分是：

1. return_sequences = False在Concat之前的图层中。
2. 如果在连接之后使用LSTM层，请return_sequences = True来填充TimeDistributed层。
3. RepeatVector层参数必须与TimeDistributed层输出中的时间步数相同。

我仅在与我的问题相关的体系结构中测试了该解决方案，因此我不确定在TimeDistributed层的其他用例中它的局限性。但是这是一个很好的解决方案，我在任何有关此问题的帖子中都找不到。