Question

过去几天我一直在调试我的NN，但我找不到问题。

我已经创建了多层感知器的全部原始实现，用于识别MNIST数据集图像。

网络似乎学习是因为在列车周期测试之后数据准确率高于94％。我有丢失功能的问题 - 一段时间后开始增加，当测试/ val精度达到~76％时。

有人可以查看我的前进/后退数学并告诉我，我的损失功能是否正确实施，或建议可能出现的问题？

NN结构：

输入层：758个节点，（每个像素1个节点）
隐藏层1：300个节点
隐藏层2：75个节点
输出图层：10个节点

NN激活功能：

输入图层 - ＆gt;隐藏层1：ReLU
隐藏层1 - ＆gt;隐藏层2：ReLU
隐藏层2 - ＆gt;输出层3：Softmax

NN损失功能：

分类交叉熵

Full CLEAN code available here as Jupyter Notebook.

神经网络前进/后退：

def train(self, features, targets):
        n_records = features.shape[0]

        # placeholders for weights and biases change values
        delta_weights_i_h1 = np.zeros(self.weights_i_to_h1.shape)
        delta_weights_h1_h2 = np.zeros(self.weights_h1_to_h2.shape)
        delta_weights_h2_o = np.zeros(self.weights_h2_to_o.shape)
        delta_bias_i_h1 = np.zeros(self.bias_i_to_h1.shape)
        delta_bias_h1_h2 = np.zeros(self.bias_h1_to_h2.shape)
        delta_bias_h2_o = np.zeros(self.bias_h2_to_o.shape)

        for X, y in zip(features, targets):
            ### forward pass
            # input to hidden 1
            inputs_to_h1_layer = np.dot(X, self.weights_i_to_h1) + self.bias_i_to_h1
            inputs_to_h1_layer_activated = self.activation_ReLU(inputs_to_h1_layer)

            # hidden 1 to hidden 2
            h1_to_h2_layer = np.dot(inputs_to_h1_layer_activated, self.weights_h1_to_h2) + self.bias_h1_to_h2
            h1_to_h2_layer_activated = self.activation_ReLU(h1_to_h2_layer)

            # hidden 2 to output
            h2_to_output_layer = np.dot(h1_to_h2_layer_activated, self.weights_h2_to_o) + self.bias_h2_to_o
            h2_to_output_layer_activated = self.softmax(h2_to_output_layer)

            # output
            final_outputs = h2_to_output_layer_activated 

            ### backpropagation
            # output to hidden2
            error = y - final_outputs
            output_error_term = error.dot(self.dsoftmax(h2_to_output_layer_activated))

            h2_error = np.dot(output_error_term, self.weights_h2_to_o.T)
            h2_error_term = h2_error * self.activation_dReLU(h1_to_h2_layer_activated)

            # hidden2 to hidden1
            h1_error = np.dot(h2_error_term, self.weights_h1_to_h2.T) 
            h1_error_term = h1_error * self.activation_dReLU(inputs_to_h1_layer_activated)

            # weight & bias step (input to hidden)
            delta_weights_i_h1 += h1_error_term * X[:, None]
            delta_bias_i_h1 = np.sum(h1_error_term, axis=0)

            # weight & bias step (hidden1 to hidden2)
            delta_weights_h1_h2 += h2_error_term * inputs_to_h1_layer_activated[:, None]
            delta_bias_h1_h2 = np.sum(h2_error_term, axis=0)

            # weight & bias step (hidden2 to output)
            delta_weights_h2_o += output_error_term * h1_to_h2_layer_activated[:, None]
            delta_bias_h2_o = np.sum(output_error_term, axis=0)

        # update the weights and biases     
        self.weights_i_to_h1 += self.lr * delta_weights_i_h1 / n_records
        self.weights_h1_to_h2 += self.lr * delta_weights_h1_h2 / n_records
        self.weights_h2_to_o += self.lr * delta_weights_h2_o / n_records
        self.bias_i_to_h1 += self.lr * delta_bias_i_h1 / n_records
        self.bias_h1_to_h2 += self.lr * delta_bias_h1_h2 / n_records
        self.bias_h2_to_o += self.lr * delta_bias_h2_o / n_records

激活功能实施：

def activation_ReLU(self, x):
    return x * (x > 0)

def activation_dReLU(self, x):
    return 1. * (x > 0)

def softmax(self, x):
    z = x - np.max(x)
    return np.exp(z) / np.sum(np.exp(z))

def dsoftmax(self, x):
    # TODO: vectorise math
    vec_len = len(x)
    J = np.zeros((vec_len, vec_len))
    for i in range(vec_len):
        for j in range(vec_len):
            if i == j:
                J[i][j] = x[i] * (1 - x[j])
            else:
                J[i][j] = -x[i] * x[j]
    return J

损失函数实现：

def categorical_cross_entropy(pred, target): 
    return (1/len(pred)) * -np.sum(target * np.log(pred))

Answer 1

我设法找到了问题。

神经网络很大，所以我无法坚持这个问题。虽然如果你查看我的Jupiter Notebook你可以看到我的Softmax激活功能的实现，以及如何在火车循环中使用它。

丢失误算的问题是由于我的Softmax实现仅适用于ndarray dim == 1 。

在训练步骤中，我只将ndarray与dim 1放在激活函数中，因此NN学得很好，但我的run()函数返回了错误的预测，因为我已经将整个测试数据插入其中，而不仅仅是单行在for循环中。因此，它计算了Softmax＆＃34;矩阵式＆＃34;而不是＆＃34;行式＆＃34;。

这是非常快速的解决方法：

   def softmax(self, x):
        # TODO: vectorise math to speed up computation
        softmax_result = None
        if x.ndim == 1:
            z = x - np.max(x)
            softmax_result = np.exp(z) / np.sum(np.exp(z))
            return softmax_result
        else:
            softmax_result = []
            for row in x:
                z = row - np.max(row)
                row_softmax_result = np.exp(z) / np.sum(np.exp(z))
                softmax_result.append(row_softmax_result)
            return np.array(softmax_result)

然而，这个代码应该被矢量化以避免for循环和ifs，如果可能的话，因为它目前很难看并且需要太多的PC资源。

神经网络损失开始增加，而acc / vol数据集的acc增加

1 个答案: