我有一个在相对较小的数据集上受过训练的模型，并且效果很好。但是现在我需要扩展，并且想使用tensorflow数据API。我使用CSVDataset，然后在输入模型之前通过地图应用了一些转换。后来，我有了一个Concatenate层，可以很好地与numpy数组配合使用，但是现在对我的数据集却失败了。
这是痕迹：
Tensorflow version: 1.12.0
Keras version: 2.1.6-tf
Initializing training dataset iterator...
Generating model...
2018-12-29 16:26:04.654201: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-12-29 16:26:04.677006: W tensorflow/core/framework/allocator.cc:122] Allocation of 80000000 exceeds 10% of system memory.
2018-12-29 16:26:04.791312: W tensorflow/core/framework/allocator.cc:122] Allocation of 80000000 exceeds 10% of system memory.
2018-12-29 16:26:04.815792: W tensorflow/core/framework/allocator.cc:122] Allocation of 80000000 exceeds 10% of system memory.
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
words_input (InputLayer) (None, 20) 0
__________________________________________________________________________________________________
casing_input (InputLayer) (None, 20) 0
__________________________________________________________________________________________________
embedding (Embedding) (None, 20, 50) 20000000 words_input[0][0]
__________________________________________________________________________________________________
embedding_1 (Embedding) (None, 20, 9) 81 casing_input[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate) (None, 20, 59) 0 embedding[0][0]
embedding_1[0][0]
__________________________________________________________________________________________________
bidirectional (Bidirectional) [(None, 400), (None, 416000 concatenate[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 100) 40100 bidirectional[0][0]
__________________________________________________________________________________________________
dense_output (Dense) (None, 1573) 158873 dense[0][0]
==================================================================================================
Total params: 20,615,054
Trainable params: 614,973
Non-trainable params: 20,000,081
__________________________________________________________________________________________________
Compiling model...
Training model...
Epoch 1/20
Traceback (most recent call last):
File "/opt/program/test-pipe.py", line 176, in <module>
model.fit(dataset, epochs=nb_epochs, steps_per_epoch=100)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training.py", line 1639, in fit
validation_steps=validation_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training_arrays.py", line 154, in fit_loop
outs = f(ins)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/backend.py", line 2986, in __call__
run_metadata=self.run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1439, in __call__
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [1,20,1,50] vs. shape[1] = [1,20,1,9]
[[{{node concatenate/concat}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding/embedding_lookup, embedding_1/embedding_lookup, concatenate/concat/axis)]]
这是我的代码的简化版本：
...
# CSV contains: text, label
dataset = tf.data.experimental.CsvDataset(['my.csv'], [tf.string, tf.int32])
def parser(text, label):
words_indexes, cases_indexes, y_oh = tf.py_func(
parse_one, [text, label], [tf.int32, tf.int32, tf.int32])
words_indexes.set_shape((20))
cases_indexes.set_shape((20))
y_oh.set_shape((1573))
return {"words_input": words_indexes, "casing_input": cases_indexes}, y_oh
dataset = dataset.map(parser).batch(1).repeat()
print("Generating model...")
words_input = Input(shape=(20, ), dtype='int32', name='words_input')
words = Embedding(
input_dim=word_to_vec_map.shape[0], output_dim=word_to_vec_map.shape[1],
weights=[word_to_vec_map], trainable=False)(words_input)
casing_input = Input(shape=(20, ), dtype='int32', name='casing_input')
casing = Embedding(
output_dim=case_embeddings.shape[1], input_dim=case_embeddings.shape[0],
weights=[case_embeddings], trainable=False)(casing_input)
output = Concatenate()([words, casing])
output = Bidirectional(
LSTM(200, return_sequences=False, return_state=True, dropout=0.50, recurrent_dropout=0.25)
)(output)
output = Dense(100, activation='relu')(output[0])
output = Dense(1573, activation='softmax', name='dense_output')(output)
model = Model(inputs=[words_input, casing_input], outputs=[output])
model.summary()
model.compile(
loss='categorical_crossentropy', optimizer='nadam', metrics=['accuracy'])
model.fit(dataset, epochs=20, steps_per_epoch=100)
您可以看到2个输入图层的大小相同，除了最后一个尺寸。因此，它应该合并（并且在将CSV加载到np数组中时也可以合并）。 这是错误吗？还是我错过了什么。非常感谢:)
编辑
出于测试目的，我只是更改了代码，以使2层具有相同的形状...现在出现此错误：
2018-12-29 16:59:14.775214: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at transpose_op.cc:157 : Invalid argument: transpose expects a vector of size 4. But input(1) is a vector of size 3
Traceback (most recent call last):
File "/opt/program/test-pipe.py", line 176, in <module>
model.fit(dataset, epochs=nb_epochs, steps_per_epoch=100)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training.py", line 1639, in fit
validation_steps=validation_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training_arrays.py", line 154, in fit_loop
outs = f(ins)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/backend.py", line 2986, in __call__
run_metadata=self.run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1439, in __call__
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: transpose expects a vector of size 4. But input(1) is a vector of size 3
[[{{node bidirectional/transpose}} = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _class=["loc:@bidirectional/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](concatenate/concat, bidirectional/transpose/perm)]]
编辑2
添加重塑图层修复了我的第二个问题。但 1 /为什么是这一层？双向LSTM应该返回正确的形状（在我之前的代码中没有使用Dataset API） 2 /出现一个新问题。这似乎有些困难，因为它似乎发生在精度指标计算中。而且我不知道这次要在哪里“重塑” ...甚至TFDBG也无法找到发生错误的“节点”。
这是模型：
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
words_input (InputLayer) (None, 20) 0
__________________________________________________________________________________________________
casing_input (InputLayer) (None, 20) 0
__________________________________________________________________________________________________
words_embedding (Embedding) (None, 20, 50) 20000000 words_input[0][0]
__________________________________________________________________________________________________
case_embedding (Embedding) (None, 20, 50) 2500 casing_input[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate) (None, 20, 100) 0 words_embedding[0][0]
case_embedding[0][0]
__________________________________________________________________________________________________
reshape (Reshape) (None, 20, 100) 0 concatenate[0][0]
__________________________________________________________________________________________________
bidirectional (Bidirectional) [(None, 400), (None, 481600 reshape[0][0]
__________________________________________________________________________________________________
dense_output (Dense) (None, 1573) 630773 bidirectional[0][0]
==================================================================================================
Total params: 21,114,873
Trainable params: 1,112,373
Non-trainable params: 20,002,500
__________________________________________________________________________________________________
这是一个隐秘的错误：
Epoch 1/20
Traceback (most recent call last):
File "/opt/program/test-pipe.py", line 194, in <module>
model.fit(dataset, epochs=nb_epochs, steps_per_epoch=10000)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training.py", line 1639, in fit
validation_steps=validation_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training_arrays.py", line 154, in fit_loop
outs = f(ins)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/backend.py", line 2986, in __call__
run_metadata=self.run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1439, in __call__
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [32,1573] vs. [32]
[[{{node metrics/acc/Equal}} = Equal[T=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](metrics/acc/ArgMax, metrics/acc/ArgMax_1)]]