Question

我正在对单节点感知器实施批量训练，但不知道如何更新我的偏差。

我正在更新权重如下：

对于每个批次，我在单个批次中运行以下临时权重更新

# update weight_update where y[i] is the actual label and o1 is the predicted output and x[i] is the input
weight_update = weight_update + (self.weights + self.learning_rate * (y[i] - o1)*x[i])

然后一个完整的批次完成我更新我班级的主要体重

# update main weights (self.weights) where len(x) is the number of samples
self.weights = self.weights + (weight_update / len(x))

Answer 1

我假设 (y[i] - o1)*x[i]) 是损失函数 wrt 权重的偏导数，我不确定你使用了什么，但假设你使用了 -1/2 * (y[i] - o1)^2

now let o1 = wx + b, where w is weight matrix and b is bias vector, 
also let, L = -1/2 * (y[i] - o1)^2
you have already calculated dLdw = dLd(o1) * d(o1)dw = (y[i] - o1) * x

In a summilar way calculate dLdb, 
dLdb = dLd(o1) * d(o1)db
dLd(o1) = (y[i] - o1)
d(o1)db = d/db (wx + b) = 0 + 1 = 1
so dLdb = (y[i] - o1) * 1

现在这一行，

weight_update = weight_update + (self.weights + self.learning_rate * (y[i] - o1)*x[i])

此时无需添加权重，只需添加渐变

weight_update += self.learning_rate * dLdw
# similarily
bias_update += self.learning_rate * dLdb

当一批完成后，就做

# update main weights (self.weights) and biases (self.biases)
# where len(x) is the number of samples
self.weights += (weight_update / len(x))
self.biases+= (bias_update / len(x)) 

# dont forget to set the values of weight_update, bias_update  to 0

This 是我几天前写的东西（MNIST 示例的 1 个隐藏层网络）您可能会发现这很有用

如何在感知器的批量训练期间计算偏差

1 个答案: