Question

在Andrew Traks's example之后，我想实现一个3层神经网络--1个输入，1个隐藏，1个输出 - 带有简单的丢失，用于二进制分类。

如果我包含偏见词b1和b2，那么我需要稍微修改一下安德鲁的代码，如下所示。

X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
y = np.array([[0,1,1,0]]).T
alpha,hidden_dim,dropout_percent = (0.5,4,0.2)
synapse_0 = 2*np.random.random((X.shape[1],hidden_dim)) - 1
synapse_1 = 2*np.random.random((hidden_dim,1)) - 1
b1 = np.zeros(hidden_dim)
b2 = np.zeros(1)
for j in range(60000):
    # sigmoid activation function
    layer_1 = (1/(1+np.exp(-(np.dot(X,synapse_0) + b1))))
    # dropout
    layer_1 *= np.random.binomial([np.ones((len(X),hidden_dim))],1-dropout_percent)[0] * (1.0/(1-dropout_percent))
    layer_2 = 1/(1+np.exp(-(np.dot(layer_1,synapse_1) + b2)))
    # sigmoid derivative = s(x)(1-s(x))
    layer_2_delta = (layer_2 - y)*(layer_2*(1-layer_2))
    layer_1_delta = layer_2_delta.dot(synapse_1.T) * (layer_1 * (1-layer_1))
    synapse_1 -= (alpha * layer_1.T.dot(layer_2_delta))
    synapse_0 -= (alpha * X.T.dot(layer_1_delta))
    b1 -= alpha*layer_1_delta
    b2 -= alpha*layer_2_delta

问题是，当然，b1维度以上的代码与layer_1_delta的维度不匹配，与b2和layer_2_delta类似。

我不明白如何计算增量b1和b2的增量 - 根据Michael Nielsen's example，b1和b2应更新为在我的代码中我认为分别为layer_1_delta和layer_2_delta的delta。

我在这里做错了什么？我是否搞砸了三角洲或偏见的维度？我觉得这是后者，因为如果我从这段代码中删除偏见就行了。提前致谢

Answer 1

首先，我要将X中的bX更改为0，将1更改为synapse_X，因为这是他们所属的地方，并且可以实现：{/ p>

b1 -= alpha * 1.0 / m * np.sum(layer_2_delta)
b0 -= alpha * 1.0 / m * np.sum(layer_1_delta)

其中m是训练集中的示例数。此外，丢弃率非常高，实际上会损害收敛。所以考虑到整个代码：

import numpy as np

X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
m = X.shape[0]
y = np.array([[0,1,1,0]]).T
alpha,hidden_dim,dropout_percent = (0.5,4,0.02)
synapse_0 = 2*np.random.random((X.shape[1],hidden_dim)) - 1
synapse_1 = 2*np.random.random((hidden_dim,1)) - 1
b0 = np.zeros(hidden_dim)
b1 = np.zeros(1)
for j in range(10000):
    # sigmoid activation function
    layer_1 = (1/(1+np.exp(-(np.dot(X,synapse_0) + b0))))
    # dropout
    layer_1 *= np.random.binomial([np.ones((len(X),hidden_dim))],1-dropout_percent)[0] * (1.0/(1-dropout_percent))
    layer_2 = 1/(1+np.exp(-(np.dot(layer_1,synapse_1) + b1)))
    # sigmoid derivative = s(x)(1-s(x))
    layer_2_delta = (layer_2 - y)*(layer_2*(1-layer_2))
    layer_1_delta = layer_2_delta.dot(synapse_1.T) * (layer_1 * (1-layer_1))
    synapse_1 -= (alpha * layer_1.T.dot(layer_2_delta))
    synapse_0 -= (alpha * X.T.dot(layer_1_delta))
    b1 -= alpha * 1.0 / m * np.sum(layer_2_delta)
    b0 -= alpha * 1.0 / m * np.sum(layer_1_delta)

print layer_2

神经网络中的反向传播偏差

1 个答案: