Question

我是tensorflow的新手，并尝试实现类似于https://www.tensorflow.org/get_started/mnist/beginners的简单单层线性网络

  x = tf.placeholder(tf.float32, [None, IN_SIZE], name="input") 

  W1 = tf.Variable(tf.zeros([IN_SIZE, OUT_SIZE]), name="Weight1") 
  b1 = tf.Variable(tf.zeros([OUT_SIZE]), name="bias1") 
  y = tf.matmul(x, W1) + b1 

  y_ = tf.placeholder(tf.float32, [None, OUT_SIZE], name="target") 

  cross_entropy = tf.reduce_mean( 
      tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y)) 
  train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy)

该程序按预期工作，我没有问题。但是，我尝试添加另一个图层，但只发现所学习的W1,b1,W2都是零矩阵，只有偏见b2包含非零值。以下是我修改过的网络

  x = tf.placeholder(tf.float32, [None, IN_SIZE], name="input") 

  W1 = tf.Variable(tf.zeros([IN_SIZE, L1_SIZE]), name="Weight1") 
  b1 = tf.Variable(tf.zeros([L1_SIZE]), name="bias1") 
  y = tf.matmul(x, W1) + b1 

  W2 = tf.Variable(tf.zeros([L1_SIZE, OUT_SIZE]), name="Weight2") 
  b2 = tf.Variable(tf.zeros([OUT_SIZE]), name="bias2") 
  y = tf.nn.relu(y) 
  y = tf.matmul(y, W2) + b2 

  # Define loss and optimizer 
  y_ = tf.placeholder(tf.float32, [None, OUT_SIZE], name="target") 

  cross_entropy = tf.reduce_mean( 
      tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y)) 
  train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy)

Answer 1

问题在于，如果在relu之前初始化权重矩阵，则渐变将始终为零，并且不会进行任何学习。你需要做随机初始化。

张量流中的优化器在非线性工作中不起作用

1 个答案: