模型拟合keras中的大数据库

时间:2019-12-10 09:07:41

标签: keras bigdata

我的模型很简单,但是数据库很大。

为解决这个问题,我将数据分为20个部分,并将数据遍历4次,总共3个时期-

attributes = [td.text for td in tr.select('td') if td.text != '']
product = Product(*attributes)

但是从新数据开始,模型获得的准确性与之相同:

batch_size = 64
num_classes = 3
epochs = 3
img_rows, img_cols,img_deep = 257, 7,7
divide=21
loops=4

for p in range(1,loops):
  for g in range(1,divide):

    ##load_train
    dataset_file=('./data_sets/dataset_%d.pickle'%g)
    label_file=('./data_sets/lebel_%d.pickle'%g)
    x = cloudpickle.load(open(dataset_file, 'rb'))
    y = cloudpickle.load(open(label_file, 'rb'))
    x_train = np.angle(x)
    y_train = y


    if K.image_data_format() == 'channels_first':
        x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols,img_deep)
    else:
        x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols,img_deep, 1)
    y_train = keras.utils.to_categorical(y_train, num_classes)

    # Split the data
    x_train, x_valid, y_train, y_valid = train_test_split(x_train, y_train, test_size=0.25, shuffle= 
    True)

    model.fit(x=x_train,y=y_train, batch_size=batch_size, epochs=epochs, verbose=1,validation_data= 
    (x_valid,y_valid),shuffle=True)

我认为这是因为fit in all循环仅引用新数据,而不是在引用此数据之前引用新数据。

感谢帮助。

0 个答案:

没有答案