Question

我有3000张（320 * 320）形状的图像，他们的拍摄时间以及他们的标签。现在我想使用这两种类型的数据（图像和时间）来预测它们的标签。主要代码如下：

num_classes=10
image_out=GlobalMaxPooling2D()(x)
x=keras.layers.advanced_activations.LeakyReLU(alpha=0.3)(x)

#add time
time_input = Input(shape=(1,), name='time_input')
x = keras.layers.concatenate([image_out, time_input],axis=-1)
x=Dense(num_classes, activation='sigmoid')(x)
predictions = Dense(num_classes,activation='softmax',name='main_output')(x)

我知道我可以使用以下代码来训练模型：

model.fit([main_input, additional_data], [labels],
      epochs=50, batch_size=32)

然而，如何使用fit_generator，因为我使用数据增量并且还使用时间特征来训练模型？我可以定义一个这样的新生成器吗？

def generator_for2img(X1, X2, Y,batch_size):
    datagen = ImageDataGenerator(featurewise_center=False,  # set input mean to 0 over the dataset
                                 samplewise_center=False,  # set each sample mean to 0
                                 featurewise_std_normalization=False,  # divide inputs by std of the dataset
                                 samplewise_std_normalization=False,  # divide each input by its std
                                 zca_whitening=False,  # apply ZCA whitening
                                 rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
                                 width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
                                 height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
                                 horizontal_flip=True,  # randomly flip images
                                 vertical_flip=False)  # randomly flip image
    time_generator=Generator()
    datagen.fit(X1)
    genX1 = datagen.flow(X1,Y,batch_size=batch_size)
    genX2 = time_generator.flow(X2,batch_size=batch_size)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i ], X1i[1]

model.fit_generator(generator_for2img(X_train_cv, X_train_time, y_train_cv,batch_size),
                                    epochs=epochs,
                                    validation_data=([X_valid,X_valid_time],y_valid),
                                    workers=4,

Answer 1

我最近遇到了一个非常类似的问题。我正在对CNN进行分类图像，但是，我还希望在分类过程中包含图像的全局特征。我预先计算了这些特征，然后希望通过后期融合（即在softmax层合并）将它们包含在网络中。我发现除了使用Keras的ImageDataGenerator之外，我还需要编写3个生成器（事实上，这可以减少到2个）。以下是包含这些生成器的功能。输入x_train和x_test是图像输入，而x_train_feat和x_test_feat表示全局（“正常”）特征。

def train_model(model, x_train, x_train_feat, y_train, x_test, x_test_feat, y_test, train_batch_size, test_batch_size, epochs, model_dir, model_name, num_classes, patience=5, monitor='val_acc'):

    '''
    Training function
    '''

    train_datagen = ImageDataGenerator(
        featurewise_center=False, # also use in test gen if activated AND fit test_gen on train data
        samplewise_center=False,
        featurewise_std_normalization=False, # also use in test gen if activated AND fit test_gen on train data
        samplewise_std_normalization=False,
        zca_whitening=False,
        zca_epsilon=0,
        rotation_range=0.05,
        width_shift_range=0.05,
        height_shift_range=0.05,
        channel_shift_range=0,
        fill_mode='nearest',
        cval=0,
        vertical_flip=False,
        rescale=1./255,
        shear_range=0.,
        zoom_range=0.,
        horizontal_flip=False)

    train_datagen.fit(x_train)

    test_datagen = ImageDataGenerator(
        rescale=1./255,
        featurewise_std_normalization=False,
        featurewise_center=False)

    #test_datagen.fit(x_train)

    train_generator = train_datagen.flow(
        x_train,
        y_train,
        batch_size=train_batch_size,
        shuffle=False)

    def train_feat_gen(x_train_feat, train_batch_size):
        while True:
            for batch in range(len(x_train_feat) // train_batch_size + 1):
                if batch > max(range(len(x_train_feat) // train_batch_size)):
                    yield x_train_feat[batch*train_batch_size:]
                else:
                    yield x_train_feat[batch*train_batch_size:(1+batch)*train_batch_size]

    def val_feat_gen(x_val_feat, test_batch_size):
        while True:
            for batch in range(len(x_val_feat) // test_batch_size + 1):
                if batch > max(range(len(x_val_feat) // test_batch_size)):
                    yield x_val_feat[batch*test_batch_size:]
                else:
                    yield x_val_feat[batch*test_batch_size:(1+batch)*test_batch_size]

    def merge_generator(gen1, gen2):
        while True:
            X1 = gen1.__next__()
            X2 = gen2.__next__()
            yield [X1[0], X2], X1[1]

    validation_generator = test_datagen.flow(
        x_test,
        y_test,
        batch_size=test_batch_size)

    final_train_gen = merge_generator(train_generator, train_feat_gen(x_train_feat, train_batch_size))
    final_val_gen = merge_generator(validation_generator, val_feat_gen(x_val_feat, test_batch_size))

    callbacks = [ModelCheckpoint(MODEL_DIR+model_name+'.h5',
                                 monitor=monitor,
                                save_best_only=True),
                EarlyStopping(monitor=monitor, patience=patience),
                TensorBoard(LOG_DIR+model_name+'_'+str(time())),
                ReduceLROnPlateau(monitor='val_loss', factor=0.75, patience=2)]

    model.fit_generator(
        final_train_gen,
        steps_per_epoch=len(x_train) // train_batch_size,
        epochs=epochs,
        validation_data=final_val_gen,
        validation_steps=len(y_test) // test_batch_size,
        callbacks=callbacks,)

请注意生成器train_feat_gen，val_feat_gen和merge_generator。前两个处理训练和验证数据的批处理，而后者包装两个生成器（train_feat_gen和train_datagen; val_feat_gen和test_datagen）以生成一个提供两个不同类型的输入的生成器。当然，同时拥有train_feat_gen和val_feat_gen是多余的，因为它们基本上是相同的，但是使用不同的参数进行实例化。但是，我认为这个案例更加明显。希望这会有所帮助。

在keras中，如何适应不同类型的多个输入数据

1 个答案: