Question

我想使用TFRecords来在ImageNet上训练模型（下面的代码）。一切都很好，并且训练进行得很好，除了验证似乎是针对训练集而不是验证集执行的。我通过将训练集简化而对验证集不做任何操作来进行测试，这导致训练和验证准确性均为100％。

当我使用fit_generator和flow_from_directory时，没有这样的问题。

这是我创建TFRecordDatasets的方法：

class ImageNetTFRecordDataset:
    def __init__(self, dir, batch_size):
        self._dir = dir
        self._batch_size = batch_size

    @staticmethod
    def _parse_function(proto):
        # read image
        keys_to_features = {'image/encoded': tf.FixedLenFeature([], tf.string),
                            'image/class/label': tf.FixedLenFeature([], tf.int64)}

        # Load one example
        parsed_features = tf.parse_single_example(proto, keys_to_features)

        # Turn your saved image string into an array
        parsed_features['image/encoded'] = tf.image.decode_jpeg(parsed_features['image/encoded'])

        parsed_features['image/encoded'] = tf.image.convert_image_dtype(parsed_features['image/encoded'], tf.float32)
        parsed_features['image/encoded'] = tf.image.per_image_standardization(parsed_features['image/encoded'])

        return parsed_features['image/encoded'], parsed_features["image/class/label"]

    def create_dataset(self):

        dir_files = os.listdir(self._dir)
        dataset = tf.data.TFRecordDataset([os.path.join(self._dir, f) for f in dir_files])

        # This dataset will go on forever
        dataset = dataset.repeat()

        # Set the number of datapoints you want to load and shuffle
        # dataset = dataset.shuffle(self._batch_size * 10)

        # Maps the parser on every filepath in the array. You can set the number of parallel loaders here
        dataset = dataset.map(self._parse_function, num_parallel_calls=8)

        # Set the batchsize
        dataset = dataset.batch(self._batch_size)

        dataset = dataset.prefetch(4)

        # Create an iterator
        iterator = dataset.make_one_shot_iterator()

        # Create your tf representation of the iterator
        image, label = iterator.get_next()

        # Bring your picture back in shape
        image = tf.reshape(image, [-1, 224, 224, 3])

        # Create a one hot array for your labels
        label = tf.one_hot(label, 1000)

        return image, label

这就是我训练模型的方式：

train_dataset = ImageNetTFRecordDataset(train_dir, BATCH_SIZE)
val_dataset = ImageNetTFRecordDataset(val_dir, BATCH_SIZE)
x, y = train_dataset.create_dataset()
x_val, y_val = val_dataset.create_dataset()

model = get_model(x)  # this model's first layer is Input(tensor=x)
model.compile(optimizer=OPTIMIZER, loss=categorical_crossentropy,
                  metrics=get_metrics(), target_tensors=[y])

model.fit(epochs=EPOCHS, verbose=1, validation_data=(x_val, y_val),
                  steps_per_epoch=150, callbacks=get_callbacks(),
                  validation_steps=VAL_STEPS)

任何想法哪里出了问题？验证部分为何会产生训练集？

验证数据导致使用TFRecordDataset在keras.fit中训练数据

0 个答案: