Question

我将训练数据和验证数据堆叠在两个张量中。首先，我使用keras.model.fit()函数运行了一个NN。出于我的目的，我希望移至keras.model.fit_generator()。我建立了一个生成器，我注意到样本数量不是批次大小的乘积。

我的实现克服了这个问题：

indices = np.arange(len(dataset))# generate indices of len(dataset)
num_of_steps = int(np.ceil(len(dataset)/batch_size)) #number of steps per epoch
extra = num_of_steps *batch_size-len(dataset)#find the size of extra samples needed to complete the next multiplication of batch_size
additional = np.random.randint(len(dataset),size = extra )#complete with random samples 
indices = np.append(indices ,additional )

在将每个时期的索引随机化之后，我简单地分批跳过此步骤，并合并正确的数据和标签。

我正在观察模型性能的下降。使用fit()进行训练时，我得到0.99的训练精度和0.93的验证精度，而使用fit_generator()进行训练时，我分别得到了0.95和0.9。请注意，这是一致的，不是单个实验。我认为这可能是由于fit()以不同的方式处理了所需的额外样本。我的实施合理吗？ fit()如何处理大小不同于batch_size乘法的数据集？

共享完整的生成器代码：

    def generator(self,batch_size,train):
        """
        Generates batches of samples
        :return: 
        """
        while 1:
            nb_of_steps=0
            if(train):    
                nb_of_steps = self._num_of_steps_train
                indices = np.arange(len(self._x_train))
                additional = np.random.randint(len(self._x_train), size=self._num_of_steps_train*batch_size-len(self._x_train))
            else:
                nb_of_steps = self._num_of_steps_test
                indices = np.arange(len(self._x_test))
                additional = np.random.randint(len(self._x_test), size=self._num_of_steps_test*batch_size-len(self._x_test))

            indices = np.append(indices,additional)
            np.random.shuffle(indices)
#            print(indices.shape)
#            print(nb_of_steps)

            for i in range(nb_of_steps):
                batch_indices=indices[i:i+batch_size]
                if(train):
                    feat = self._x_train[batch_indices]
                    label = self._y_train[batch_indices]
                else:
                    feat = self._x_test[batch_indices]
                    label = self._y_test[batch_indices]
                feat = np.expand_dims(feat,axis=1)
#                print(feat.shape)
#                print(label.shape)

                yield feat, label

Answer 1

看起来您可以大大简化生成器！步数等可以在循环外部设置，因为它们并没有真正改变。此外，看起来batch_indices并没有遍历整个数据集。最后，如果您的数据适合存储在内存中，则可能根本不需要生成器，但是您可以自行决定。

def generator(self, batch_size, train):
    nb_of_steps = 0
    if (train):
        nb_of_steps = self._num_of_steps_train
        indices = np.arange(len(self._x_train)) #len of entire dataset
    else:
        nb_of_steps = self._num_of_steps_test
        indices = np.arange(len(self._x_test))
    while 1:
        np.random.shuffle(indices)
        for i in range(nb_of_steps):
            start_idx = i*batch_size
            end_idx = min(i*batch_size+batch_size, len(indices))
            batch_indices=indices[start_idx : end_idx]
            if(train):
                feat = self._x_train[batch_indices]
                label = self._y_train[batch_indices]
            else:
                feat = self._x_test[batch_indices]
                label = self._y_test[batch_indices]
            feat = np.expand_dims(feat,axis=1)
            yield feat, label

对于更强大的生成器，请考虑使用keras.utils.Sequence class为您的集合创建一个类。它将添加一些额外的代码行，但是肯定可以与keras一起使用。

Keras Fit与Fit_generator额外的枫叶

1 个答案: