Tensorflow融合操作(例如BatchNorm,LRN,sparse_softmax_cross_entropy_with_logits

时间:2018-10-09 06:58:08

标签: python tensorflow gradients

当前,如果有人在TensorFlow中定义了一个包含BatchNorm或LRN的网络,并尝试添加一个损失项,该损失项包含了相对于输入的交叉熵的梯度,则在以下几行将引发错误:

LookupError: No gradient defined for operation 'clone_0/gradients/clone_0/InceptionV3/InceptionV3/Conv2d_1a_3x3/BatchNorm/FusedBatchNorm_grad/FusedBatchNormGrad' (op type: FusedBatchNormGrad)

如果我们在网络中有一个tf.nn.lrn层,则会引发类似的情况:

LookupError: gradient registry has no entry for: LRNGrad
During handling of the above exception, another exception occurred:
...................
LookupError: No gradient defined for operation 'gradients/norm1_grad/LRNGrad' (op type: LRNGrad). 

链接到LRN的github问题-https://github.com/tensorflow/tensorflow/issues/22825

对于tf.nn.sparse_softmax_cross_entropy_with_logits操作,也缺少二阶梯度,并且出现以下错误:

LookupError: Gradient explicitly disabled. Reason: b"Currently there is no way to take the second derivative of sparse_softmax_cross_entropy_with_logits due to the fused implementation's interaction with tf.gradients()" . 

链接到sparse_softmax_cross_entropy_with_logits的github问题-https://github.com/tensorflow/tensorflow/issues/11626

这些错误表明,融合操作不支持高阶导数。我假设所有张量流操作都支持高阶导数计算。不是吗?

0 个答案:

没有答案