Question

我目前正在尝试训练模型，并且从此答案here开始，我的输入管道已构建。我想在每个时期后保存我的模型。但是经过一段时间的训练之后，训练崩溃了。我已经读到它是因为它将输入作为常数张量添加到图形中。建议使用tf.placeholder来解决问题的解决方案here。不幸的是，这并不能解决我的问题。输入管道如下所示

....
filenames = [P_1]
dataset = tf.data.TFRecordDataset(filenames)
def _parse_function(example_proto):
       keys_to_features = { 'data':tf.VarLenFeature(tf.float32)},
       parsed_features = tf.parse_single_example(example_proto,  keys_to_features)
       return tf.sparse_tensor_to_dense(parsed_features['data'
# Parse the record into tensors.
dataset = dataset.map(_parse_function)
# Shuffle the dataset
dataset = dataset.shuffle(buffer_size=1000)
# Repeat the input indefinitly 
dataset = dataset.repeat()      
# Generate batches     
dataset = dataset.batch(Batch_size) 
# Create a one-shot iterator
iterator = dataset.make_one_shot_iterator()
data = iterator.get_next()   
....
for i in range(epochs):
    for ii in range(iteration):
        image = sess.run(data)
        ....
     saver.save(sess, 'filename')

错误消息如下所示

[libprotobuf FATAL external/protobuf_archive/src/google/protobuf/message_lite.cc:68] CHECK failed: (byte_size_before_serialization) == (byte_size_after_serialization): tensorflow.GraphDef was modified concurrently during serialization.
terminate called after throwing an instance of 'google::protobuf::FatalException'  
what():  CHECK failed: (byte_size_before_serialization) == (byte_size_after_serialization): tensorflow.GraphDef was modified concurrently during serialization.
Aborted

Answer 1

问题似乎出在_parse_function中。创建TFrecord文件时，请确保解析器以相同的方式进行操作。例如，如果它们具有相同的数据类型

由于保存模型而导致训练崩溃：“ tensorflow.GraphDef在序列化过程中被同时修改”

1 个答案: