Tensorflow tf.nn.softmax()函数比手写softmax

时间:2018-03-25 03:57:15

标签: python tensorflow softmax

我正在用张量流写一个简单的逻辑回归。我发现当使用tf.nn.softmax时,算法收敛得更快,最终精度更高。 如果切换到我自己的softmax实现,网络收敛速度较慢,结束准确性不是很好。

以下是代码:

SEED = 1025
W = tf.Variable(tf.truncated_normal([image_size * image_size, num_labels], seed=SEED))
b = tf.Variable(tf.zeros([num_labels]))
logits = tf.matmul(train_dataset, W) + b

# My softmax:
y_ = tf.exp(logits) / tf.reduce_sum(tf.exp(logits), axis=0)
# Tensorflow softmax: 
y_ = tf.nn.softmax(logits)

y_clipped = tf.clip_by_value(y_, 1e-10, 0.9999999)
loss = -tf.reduce_mean(tf.reduce_sum(train_labels * tf.log(y_clipped), axis=1))

使用我的softmax:

Loss at step 0: 22.213934
Training accuracy: 12.7%
Validation accuracy: 13.2%
Loss at step 100: 12.777291
Training accuracy: 45.3%
Validation accuracy: 45.5%
Loss at step 200: 11.361242
Training accuracy: 48.2%
Validation accuracy: 47.4%
Loss at step 300: 10.658278
Training accuracy: 51.4%
Validation accuracy: 49.7%
Loss at step 400: 9.297832
Training accuracy: 59.2%
Validation accuracy: 56.8%
Loss at step 500: 8.902699
Training accuracy: 62.0%
Validation accuracy: 59.2%
Loss at step 600: 8.681184
Training accuracy: 64.2%
Validation accuracy: 61.0%
Loss at step 700: 8.529438
Training accuracy: 65.8%
Validation accuracy: 62.3%
Loss at step 800: 8.416442
Training accuracy: 66.8%
Validation accuracy: 63.3%
Test accuracy: 70.4%

使用tensorflow的softmax:

Loss at step 0: 13.555875
Training accuracy: 12.7%
Validation accuracy: 14.5%
Loss at step 100: 2.194562
Training accuracy: 72.5%
Validation accuracy: 72.0%
Loss at step 200: 1.808641
Training accuracy: 75.5%
Validation accuracy: 74.5%
Loss at step 300: 1.593390
Training accuracy: 76.8%
Validation accuracy: 75.0%
Loss at step 400: 1.442661
Training accuracy: 77.7%
Validation accuracy: 75.2%
Loss at step 500: 1.327751
Training accuracy: 78.2%
Validation accuracy: 75.4%
Loss at step 600: 1.236314
Training accuracy: 78.5%
Validation accuracy: 75.6%
Loss at step 700: 1.161479
Training accuracy: 78.9%
Validation accuracy: 75.6%
Loss at step 800: 1.098717
Training accuracy: 79.4%
Validation accuracy: 75.8%
Test accuracy: 83.3%

documentation开始,理论上张量流量的softmax应该和我实现的完全相同,不是吗?

  

此功能执行等效的

     

softmax = tf.exp(logits)/ tf.reduce_sum(tf.exp(logits),axis)

编辑:我在从正态分布初始化时添加了种子,现在我可以重现准确性结果。 在"我的softmax"中设置轴值时线,只有轴= 0不会导致错误。设置轴= 1或轴= -1都会导致此错误:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimensions must be equal, but are 10 and 10000 for 'truediv' (op: 'RealDiv') with input shapes: [10000,10], [10000].

3 个答案:

答案 0 :(得分:2)

  • 假设您的softmax实现正确
  • 首先,将tensorflow softmax与手写softmax进行比较是不公平的,因为程序中包含随机性
  • 我的意思是行W = tf.Variable(tf.truncated_normal([image_size * image_size, num_labels]))在你的程序中引入了随机性,因为权重是随机设置的,因此每次运行程序时都会得到不同的结果
  • 如果你有某种种子(某种起点),你只能比较softmax
  • 现在,如果您多次执行上述实验,并且每次tensorflow softmax击败手写softmax,那么您的问题是有效的
  • tf.truncated_normal函数确实采用种子参数...您可以使用该参数并查看结果是什么
  • 无论如何,如果你的手写softmax是正确的,那么使用种子,tensorflow softmax和你的softmax应该输出相同的结果
  • 甚至我认为你的轴在你的情况下应该是1,这是最后一个轴,因为softmax应该沿着有类的轴

答案 1 :(得分:1)

您正在将axis=0传递给"您的" SOFTMAX。虽然我不知道您的数据看起来如何,但0通常是批处理轴,沿此轴执行softmax是不正确的。请参阅tf.nn.softmax的文档:axis的默认值为-1。通常,axis应该是包含不同类的维度。

答案 2 :(得分:0)

总而言之,以下实现有效。您可以通过MNIST beginner example运行此命令,并获得相同的准确性。

# My softmax:
y1 = tf.exp(logits)
y_ = y1 / tf.reduce_sum(y1, keepdims=True, axis=1)