我想训练一个CNN(使用我自己的张量流图),但是我拥有的数据量远远超过了我可以放入一个numpy数组(馈送到CNN中)的数据。是否可以运行一个循环,在该循环中我处理不同的训练数据,并将其提供给同一会话进行训练?还是仅有效地训练了最后给出的数据?
这是我打算做的事情:
for i in range(MULTIPLE_TRAINING):
X_train, y_train, NUM_STAN = parse_data('gdrive/My Drive/Colab Notebooks/fanta/partial_data',
BONE_WINDOW, 24,start_point=i * 24, randomize=False,
should_sample=True, num_samples = 32, b_check=True, training=True)
y_out = simple_model(X, y, is_training)
total_loss = tf.losses.softmax_cross_entropy(tf.one_hot(y, 2), logits=y_out)
mean_loss = tf.reduce_mean(total_loss)
optimizer = tf.train.RMSPropOptimizer(1e-3)
train_step = optimizer.minimize(mean_loss)
# batch normalization in tensorflow requires this extra dependency
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(extra_update_ops):
train_step = optimizer.minimize(mean_loss)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print('Training')
run_model(sess,y_out,mean_loss,X_train,y_train,3,64,100,train_step,True,num_stan=NUM_STAN)
非常感谢您的帮助!目标是能够训练大量数据。