TensorFlow：训练for循环中的每次迭代都较慢

Question

我在TensorFlow中训练一个标准的，简单的多层感知器神经网络，其中有三个隐藏层。我添加了一个文本进度条，以便我可以观察迭代时代的进度。我发现的是每次迭代的处理时间在前几个时期之后增加。这是一个示例屏幕截图，显示每次迭代的增加：

在这种情况下，前几次迭代大约需要1.05s / it，而100％则需要4.01s / it。

此处列出了相关代码：

# ------------------------- Build the TensorFlow Graph -------------------------

with tf.Graph().as_default():

    (a bunch of statements for specifying the graph)

# --------------------------------- Training ----------------------------------

    sess = tf.InteractiveSession()
    sess.run(tf.initialize_all_variables())

    print "Start Training"

    pbar = tqdm(total = training_epochs)
    for epoch in range(training_epochs):
        avg_cost = 0.0
    batch_iter = 0

    while batch_iter < batch_size:
        train_features = []
        train_labels = []
        batch_segments = random.sample(train_segments, 20)
        for segment in batch_segments:
            train_features.append(segment[0])
            train_labels.append(segment[1])
        sess.run(optimizer, feed_dict={x: train_features, y_: train_labels})
        line_out = "," + str(batch_iter) + "\n"
        train_outfile.write(line_out)
        line_out = ",," + str(sess.run(tf.reduce_mean(weights['h1']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h2']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h3']), feed_dict={x: train_features, y_: train_labels})) + "\n"
        train_outfile.write(line_out)
        avg_cost += sess.run(cost, feed_dict={x: train_features, y_: train_labels})/batch_size

        batch_iter += 1

    pbar.update(1)  # Increment the progress bar by one

train_outfile.close()
print "Completed training"

在搜索stackoverflow时，我发现Processing time gets longer and longer after each iteration其他人也遇到了问题，每次迭代的时间都比上一次长。但是，我相信我的可能会有所不同，因为他们使用如下语句清楚地将操作添加到图表中：

distorted_image = tf.image.random_flip_left_right(image_tensor)

虽然我是TensorFlow的新手，但我不相信我犯了同样的错误，因为我的循环中唯一的东西是sess.run（）调用。

非常感谢任何帮助。

Answer 1

你拥有的三个地方：

sess.run(tf.reduce_mean(weights['h1']), ...)

每个节点在while循环的每次迭代中向图形添加一个新的tf.reduce_mean()节点，这会增加开销。尝试在while循环之外创建它们：

with tf.Graph().as_default():
  ...
  m1 = tf.reduce_mean(weights['h1'])

while batch_iter < batch_size:
  ...
  line_out = ",," + str(sess.run(m1, feed_dict={x: train_features, y_: train_labels}))