Question

我从TensorFlow教程开始，使用单层前馈神经网络对mnist数据集中的图像进行分类。这工作正常，我在测试集上得到80％以上。然后我尝试通过在两者之间添加一个新层来将其修改为多层网络。在此修改之后，我训练网络的所有尝试都失败了。网络的前几次迭代变得更好，但随后它停滞在11.35％的准确度。

使用1个隐藏层的前20次迭代：

Train set: 0.124, test set: 0.098
Train set: 0.102, test set: 0.098
Train set: 0.112, test set: 0.101
Train set: 0.104, test set: 0.101
Train set: 0.092, test set: 0.101
Train set: 0.128, test set: 0.1135
Train set: 0.12, test set: 0.1135
Train set: 0.114, test set: 0.1135
Train set: 0.108, test set: 0.1135
Train set: 0.1, test set: 0.1135
Train set: 0.114, test set: 0.1135
Train set: 0.11, test set: 0.1135
Train set: 0.122, test set: 0.1135
Train set: 0.102, test set: 0.1135
Train set: 0.12, test set: 0.1135
Train set: 0.106, test set: 0.1135
Train set: 0.102, test set: 0.1135
Train set: 0.116, test set: 0.1135
Train set: 0.11, test set: 0.1135
Train set: 0.124, test set: 0.1135

我训练它的时间并不重要，它被困在这里。我试图从矫正的线性单位改为softmax，两者都产生相同的结果。我试图将适应度函数改为e =（y_true-y）^ 2。结果相同。

前20次迭代不使用隐藏层：

Train set: 0.124, test set: 0.098
Train set: 0.374, test set: 0.3841
Train set: 0.532, test set: 0.5148
Train set: 0.7, test set: 0.6469
Train set: 0.746, test set: 0.7732
Train set: 0.786, test set: 0.8
Train set: 0.788, test set: 0.7887
Train set: 0.752, test set: 0.7882
Train set: 0.84, test set: 0.8138
Train set: 0.85, test set: 0.8347
Train set: 0.806, test set: 0.8084
Train set: 0.818, test set: 0.7917
Train set: 0.85, test set: 0.8063
Train set: 0.792, test set: 0.8268
Train set: 0.812, test set: 0.8259
Train set: 0.774, test set: 0.8053
Train set: 0.788, test set: 0.8522
Train set: 0.812, test set: 0.8131
Train set: 0.814, test set: 0.8638
Train set: 0.778, test set: 0.8604

这是我的代码：

import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Parameters
batch_size = 500

# Create the network structure
# ----------------------------

# First layer
x = tf.placeholder(tf.float32, [None, 784])
W_1 = tf.Variable(tf.zeros([784,10]))
b_1 = tf.Variable(tf.zeros([10]))
y_1 = tf.nn.relu(tf.matmul(x,W_1) + b_1)

# Second layer
W_2 = tf.Variable(tf.zeros([10,10]))
b_2 = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(y_1,W_2) + b_2)

# Loss function
y_true = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y), reduction_indices=[1]))

# Training method
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Train network
# -------------
sess = tf.Session()
sess.run(tf.initialize_all_variables())
batch, batch_labels = mnist.train.next_batch(batch_size)
for i in range(20):
    print("Train set: " + str(sess.run(accuracy, feed_dict={x: batch, y_true: batch_labels}))
            + ", test set: " + str(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))) 
    sess.run(train_step, feed_dict={x: batch, y_true: batch_labels})
    batch, batch_labels = mnist.train.next_batch(batch_size)

所以使用此代码它不起作用，但如果我从

更改

y = tf.nn.softmax(tf.matmul(y_1,W_2) + b_2)

到

y = tf.nn.softmax(tf.matmul(x,W_1) + b_1)

然后它的工作原理。我错过了什么？

编辑：现在我有了它的工作。需要进行两项更改，首先将权重启动为随机值而不是零（是的，实际上是权重需要不为零，尽管有relu函数，但偏差为零仍然正常）。第二件事对我来说很奇怪：如果我从输出层中删除softmax函数而不是手动应用交叉熵的公式，则使用softmax_cross_entropy_with_logits（y，y_true）函数然后它可以工作。据我所知，它应该是相同的..之前我也尝试了平方误差的总和，但也没有工作..无论如何，以下代码正在运行。（虽然相当丑陋，但工作......）通过10k迭代，它在测试集上获得了93.59％的准确度，因此不是最佳的，但比没有隐藏层的更好。经过20次迭代后，它已经达到了65％。

import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Parameters
batch_size = 500

# Create the network structure
# ----------------------------

# First layer
x = tf.placeholder(tf.float32, [None, 784])
W_1 = tf.Variable(tf.truncated_normal([784,10], stddev=0.1))
b_1 = tf.Variable(tf.truncated_normal([10], stddev=0.1))
y_1 = tf.nn.relu(tf.matmul(x,W_1) + b_1)

# Second layer
W_2 = tf.Variable(tf.truncated_normal([10,10], stddev=0.1))
b_2 = tf.Variable(tf.truncated_normal([10], stddev=0.1))
y = tf.matmul(y_1,W_2) + b_2

# Loss function
y_true = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y,y_true))

# Training method
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Train network
# -------------
sess = tf.Session()
sess.run(tf.initialize_all_variables())
batch, batch_labels = mnist.train.next_batch(batch_size)
for i in range(10000):
    if i % 100 == 0:
        print("Train set: " + str(sess.run(accuracy, feed_dict={x: batch, y_true: batch_labels}))
                + ", test set: " + str(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))) 
    sess.run(train_step, feed_dict={x: batch, y_true: batch_labels})
    batch, batch_labels = mnist.train.next_batch(batch_size)

Answer 1

很少有建议：

1-为重量变量初始化添加标准偏差，而不是使用zeros初始化：

weight_1 = tf.Variable(tf.truncated_normal([784,10], stddev=0.1))

2-降低学习率，直到准确度值显示出不同的行为。

3-使用RELU时，使用略微正值初始化偏差。这个建议可能与您所看到的问题关系不大。

bias_1 = tf.Variable(tf.constant(.05, shape=[10]))

多层前馈网无法在TensorFlow中训练

1 个答案: