Question

我正在尝试使用Keras一段时间后学习TensorFlow，我正在尝试构建一个用于CIFAR-10分类的ConvNet。但是，我认为我在TensorFlow API中误解了一些内容，因为即使在单层网络模型中权重也不会更新。

该模型的代码如下：

num_epochs = 10 
batch_size = 64

# Shape of mu and std is correct: (1, 32, 32, 3)
mu = np.mean(X_train, axis=0, keepdims=True)
sigma = np.std(X_train, axis=0, keepdims=True)

# Placeholders for data & normalization
# (normalisation does not help)
data = tf.placeholder(np.float32, shape=(None, 32, 32, 3), name='data')
labels = tf.placeholder(np.int32, shape=(None,), name='labels')
data = (data - mu) / sigma

# flatten
flat = tf.reshape(data, shape=(-1, 32 * 32 * 3))
dense1 = tf.layers.dense(inputs=flat, units=10)
predictions = tf.nn.softmax(dense1)

onehot_labels = tf.one_hot(indices=labels, depth=10)

# Tried sparse_softmax_cross_entropy_with_logits as well
loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=predictions)
loss = tf.reduce_mean(loss)

# Learning rate does not matter as the weights are not updating!
optimizer = tf.train.AdamOptimizer(learning_rate=0.01).minimize(loss)
loss_history = []

with tf.Session() as session:
    tf.global_variables_initializer().run()
    tf.local_variables_initializer().run()

    for epochs in range(10):
        print("Epoch:", epochs)
        # Load tiny batches-
        for batch in iterate_minibatches(X_train.astype(np.float32)[:10], y_train[:10], 5):
            inputs, target = batch
            feed_dict = {data: inputs, labels: target}
            loss_val, _ = session.run([loss, optimizer], feed_dict=feed_dict)
            grads = tf.reduce_sum(tf.gradients(loss, dense1)[0])
            grads = session.run(grads, {data: inputs, labels: target})
            print("Loss:", loss_val, "Grads:", grads)

代码产生以下输出：

Epoch: 0
Loss: 2.46115 Grads: -1.02031e-17
Loss: 2.46041 Grads: 0.0
Epoch: 1
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 2
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 3
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 4
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 5
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 6
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 7
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 8
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 9
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0

看起来模型可能会以某种方式重置其权重或完全停止学习。我也尝试过稀疏的softmax交叉熵损失，但没有任何帮助。

Answer 1

您正在将{soft}两次应用于输出，一次应用于tf.nn.softmax，然后再应用softmax_cross_entropy。这可能会破坏网络中的任何学习能力。

Tensorflow梯度为0，权重不更新

1 个答案: