Keras验证丢失计算错误

Question

我的模型在CPU机器上运行良好，但是当我尝试在我们的集群上重新运行它时（使用单个GPU和相同的数据集），我遇到了一个问题。当GPU机器上的训练验证损失和准确性从纪元到时期没有改善时（见下文）。在CPU机器上并非如此（我在20个时期之后能够达到验证准确度~0.8）

详细信息：

Keras 2.1.3

TensforFlow后端

70/20/10 train / dev / test

~7000张图片

模型基于ResNet50

代码

    import sys
    import math
    import os
    import glob

    from keras import applications
    from keras.preprocessing.image import ImageDataGenerator
    from keras import optimizers
    from keras.models import Sequential, Model 
    from keras.layers import Flatten, Dense
    from keras import backend as k 
    from keras.callbacks import ModelCheckpoint, CSVLogger, EarlyStopping

    ############ Training parameters ##################
    img_width, img_height = 224, 224
    batch_size = 32
    epochs = 100


    ############ Define the data ##################
    train_data_dir = '/mnt/data/train'
    validation_data_dir = '/mnt/data/validate'

    train_data_dir_class1 = os.path.join(train_data_dir,'class1', '*.jpg')
    train_data_dir_class2 = os.path.join(train_data_dir, 'class2', '*.jpg')

    validation_data_dir_class1 = os.path.join(validation_data_dir, 'class1', '*.jpg')
    validation_data_dir_class2 = os.path.join(validation_data_dir, 'class2', '*.jpg')

    # number of training and validation samples
    nb_train_samples = len(glob.glob(train_data_dir_class1)) + len(glob.glob(train_data_dir_class2))
    nb_validation_samples = len(glob.glob(validation_data_dir_class1)) + len(glob.glob(validation_data_dir_class2))


    ############ Define the model ##################
    model = applications.resnet50.ResNet50(weights = "imagenet",
                                           include_top = False,
                                           input_shape = (img_width, img_height, 3))

    for layer in model.layers:
        layer.trainable = False

    # Adding a FC layer
    x = model.output
    x = Flatten()(x)
    predictions = Dense(1, activation = "sigmoid")(x)

    # creating the final model 
    model_final = Model(inputs = model.input, outputs = predictions)

    # compile the model 
    model_final.compile(loss = "binary_crossentropy",
                        optimizer = optimizers.Adam(lr = 0.001,
                                                    beta_1 = 0.9,
                                                    beta_2 = 0.999,
                                                    epsilon = 1e-10),
                        metrics = ["accuracy"])

    # train and test generators 
    train_datagen = ImageDataGenerator(rescale = 1./255,
                                       horizontal_flip = True,
                                       fill_mode = "nearest",
                                       zoom_range = 0.3,
                                       width_shift_range = 0.3,
                                       height_shift_range = 0.3,
                                       rotation_range = 30)

    test_datagen = ImageDataGenerator(rescale = 1./255)

    train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                        target_size = (img_height, img_width),
                                                        batch_size = batch_size,
                                                        class_mode = "binary",
                                                        seed = 2018)

    validation_generator = test_datagen.flow_from_directory(validation_data_dir,
                                                            target_size = (img_height, img_width),
                                                            class_mode = "binary",
                                                            seed = 2018)

    early = EarlyStopping(monitor = 'val_loss', min_delta = 10e-5, patience = 10, verbose = 1, mode = 'auto')
    performance_log = CSVLogger('/mnt/results/vanilla_model_log.csv', separator = ',', append = False)

    # Train the model
    model_final.fit_generator(generator = train_generator,
                              steps_per_epoch = math.ceil(train_generator.samples / batch_size),
                              epochs = epochs,
                              validation_data = validation_generator,
                              validation_steps = math.ceil(validation_generator.samples / batch_size),
                              callbacks = [early, performance_log])

    # Save the model
    model_final.save('/mnt/results/vanilla_model.h5')

培训日志

Epoch 1/100
151/151 [==============================] - 237s 2s/step - loss: 0.7234 - acc: 0.5240 - val_loss: 0.9899 - val_acc: 0.5425
Epoch 2/100
151/151 [==============================] - 65s 428ms/step - loss: 0.6491 - acc: 0.6228 - val_loss: 1.0248 - val_acc: 0.5425
Epoch 3/100
151/151 [==============================] - 65s 429ms/step - loss: 0.6091 - acc: 0.6648 - val_loss: 1.0377 - val_acc: 0.5425
Epoch 4/100
151/151 [==============================] - 64s 426ms/step - loss: 0.5829 - acc: 0.6968 - val_loss: 1.0459 - val_acc: 0.5425
Epoch 5/100
151/151 [==============================] - 64s 427ms/step - loss: 0.5722 - acc: 0.7070 - val_loss: 1.0472 - val_acc: 0.5425
Epoch 6/100
151/151 [==============================] - 64s 427ms/step - loss: 0.5582 - acc: 0.7166 - val_loss: 1.0501 - val_acc: 0.5425
Epoch 7/100
151/151 [==============================] - 64s 424ms/step - loss: 0.5535 - acc: 0.7188 - val_loss: 1.0492 - val_acc: 0.5425
Epoch 8/100
151/151 [==============================] - 64s 426ms/step - loss: 0.5377 - acc: 0.7287 - val_loss: 1.0209 - val_acc: 0.5425
Epoch 9/100
151/151 [==============================] - 64s 425ms/step - loss: 0.5328 - acc: 0.7368 - val_loss: 1.0062 - val_acc: 0.5425
Epoch 10/100
151/151 [==============================] - 65s 432ms/step - loss: 0.5296 - acc: 0.7381 - val_loss: 1.0016 - val_acc: 0.5425
Epoch 11/100
151/151 [==============================] - 65s 430ms/step - loss: 0.5231 - acc: 0.7419 - val_loss: 1.0021 - val_acc: 0.5425

由于我能够在CPU机器上获得良好的结果，我假设必须在每个时代结束时错误地计算验证损失/准确度。为了测试这个理论，我使用列车集作为验证集：如果正确计算验证损失/准确度，我们应该看到大致相同的列车和验证损失和准确度值。如您所见，验证损失值与训练损失值不同，这使我相信在每个时期结束时错误地计算验证损失。 为什么会这样？有哪些可能的解决方案？

修改后的代码

    import sys
    import math
    import os
    import glob

    from keras import applications
    from keras.preprocessing.image import ImageDataGenerator
    from keras import optimizers
    from keras.models import Sequential, Model 
    from keras.layers import Flatten, Dense
    from keras import backend as k 
    from keras.callbacks import ModelCheckpoint, CSVLogger, EarlyStopping

    ############ Training parameters ##################
    img_width, img_height = 224, 224
    batch_size = 32
    epochs = 100


    ############ Define the data ##################
    train_data_dir = '/mnt/data/train'
    validation_data_dir = '/mnt/data/train' # redefined validation set to test accuracy of validation loss/accuracy calculations

    train_data_dir_class1 = os.path.join(train_data_dir,'class1', '*.jpg')
    train_data_dir_class2 = os.path.join(train_data_dir, 'class2', '*.jpg')

    validation_data_dir_class1 = os.path.join(validation_data_dir, 'class1', '*.jpg')
    validation_data_dir_class2 = os.path.join(validation_data_dir, 'class2', '*.jpg')

    # number of training and validation samples
    nb_train_samples = len(glob.glob(train_data_dir_class1)) + len(glob.glob(train_data_dir_class2))
    nb_validation_samples = len(glob.glob(validation_data_dir_class1)) + len(glob.glob(validation_data_dir_class2))


    ############ Define the model ##################
    model = applications.resnet50.ResNet50(weights = "imagenet",
                                           include_top = False,
                                           input_shape = (img_width, img_height, 3))

    for layer in model.layers:
        layer.trainable = False

    # Adding a FC layer
    x = model.output
    x = Flatten()(x)
    predictions = Dense(1, activation = "sigmoid")(x)

    # creating the final model 
    model_final = Model(inputs = model.input, outputs = predictions)

    # compile the model 
    model_final.compile(loss = "binary_crossentropy",
                        optimizer = optimizers.Adam(lr = 0.001,
                                                    beta_1 = 0.9,
                                                    beta_2 = 0.999,
                                                    epsilon = 1e-10),
                        metrics = ["accuracy"])

    # train and test generators 
    train_datagen = ImageDataGenerator(rescale = 1./255,
                                       horizontal_flip = True,
                                       fill_mode = "nearest",
                                       zoom_range = 0.3,
                                       width_shift_range = 0.3,
                                       height_shift_range = 0.3,
                                       rotation_range = 30)

    test_datagen = ImageDataGenerator(rescale = 1./255)

    train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                        target_size = (img_height, img_width),
                                                        batch_size = batch_size,
                                                        class_mode = "binary",
                                                        seed = 2018)

    validation_generator = test_datagen.flow_from_directory(validation_data_dir,
                                                            target_size = (img_height, img_width),
                                                            class_mode = "binary",
                                                            seed = 2018)

    early = EarlyStopping(monitor = 'val_loss', min_delta = 10e-5, patience = 10, verbose = 1, mode = 'auto')
    performance_log = CSVLogger('/mnt/results/vanilla_model_log.csv', separator = ',', append = False)

    # Train the model
    model_final.fit_generator(generator = train_generator,
                              steps_per_epoch = math.ceil(train_generator.samples / batch_size),
                              epochs = epochs,
                              validation_data = validation_generator,
                              validation_steps = math.ceil(validation_generator.samples / batch_size),
                              callbacks = [early, performance_log])

    # Save the model
    model_final.save('/mnt/results/vanilla_model.h5')

修改后代码的培训日志：

Epoch 1/100
151/151 [==============================] - 251s 2s/step - loss: 0.6804 - acc: 0.5910 - val_loss: 0.6923 - val_acc: 0.5469
Epoch 2/100
151/151 [==============================] - 87s 578ms/step - loss: 0.6258 - acc: 0.6523 - val_loss: 0.6938 - val_acc: 0.5469
Epoch 3/100
151/151 [==============================] - 88s 580ms/step - loss: 0.5946 - acc: 0.6874 - val_loss: 0.7001 - val_acc: 0.5469
Epoch 4/100
151/151 [==============================] - 88s 580ms/step - loss: 0.5718 - acc: 0.7086 - val_loss: 0.7036 - val_acc: 0.5469
Epoch 5/100
151/151 [==============================] - 87s 578ms/step - loss: 0.5634 - acc: 0.7157 - val_loss: 0.7067 - val_acc: 0.5469
Epoch 6/100
151/151 [==============================] - 87s 578ms/step - loss: 0.5467 - acc: 0.7243 - val_loss: 0.7099 - val_acc: 0.5469
Epoch 7/100
151/151 [==============================] - 87s 578ms/step - loss: 0.5392 - acc: 0.7317 - val_loss: 0.7096 - val_acc: 0.5469
Epoch 8/100
151/151 [==============================] - 87s 578ms/step - loss: 0.5287 - acc: 0.7387 - val_loss: 0.7083 - val_acc: 0.5469
Epoch 9/100
151/151 [==============================] - 87s 575ms/step - loss: 0.5306 - acc: 0.7385 - val_loss: 0.7088 - val_acc: 0.5469
Epoch 10/100
151/151 [==============================] - 87s 577ms/step - loss: 0.5303 - acc: 0.7318 - val_loss: 0.7111 - val_acc: 0.5469
Epoch 11/100
151/151 [==============================] - 87s 578ms/step - loss: 0.5157 - acc: 0.7474 - val_loss: 0.7143 - val_acc: 0.5469

Answer 1

一个可能有帮助的非常快速的想法。

我认为图像标签由两个图像数据生成器随机分配并经过培训。 两个图像数据生成器给出不同的标签分布。 这就是为什么训练准确度提高，而验证设置保持在50％左右。

我还没有完全检查数据图像生成器的文档。希望这可能有所帮助。

flow_from_directory（）的参数类描述了一种设置培训标签的方法。

  

classes：类子目录的可选列表（例如['dogs'，   '猫']）。默认值：无。如果未提供，则列表将为   从子目录名称/结构下自动推断出   目录，其中每个子目录将被视为不同的   class（以及将映射到标签的类的顺序   索引，将是字母数字）。包含映射的字典   从类名到类索引可以通过属性获得   class_indices。