Question

我试图将我的神经网络从隐藏和输出层的sigmoid激活更改为tanh函数。我很困惑我应该改变什么。只是神经元的输出计算或反向传播的误差计算？这是输出计算：

public void calcOutput() 
{
    if (!isBias) 
    {
        float sum = 0;
        float bias = 0;
        //System.out.println("Looking through " + connections.size() + " connections");
        for (int i = 0; i < connections.Count; i++) 
        {
            Connection c = (Connection) connections[i];
            Node from = c.getFrom();
            Node to = c.getTo();
            // Is this connection moving forward to us
            // Ignore connections that we send our output to
            if (to == this) 
            {
                // This isn't really necessary
                // But I am treating the bias individually in case I need to at some point
                if (from.isBias) bias = from.getOutput()*c.getWeight();
                else sum += from.getOutput()*c.getWeight();
            }
        }
        // Output is result of sigmoid function
        output = Tanh(bias+sum);
    }
}

它非常适合我以前训练它的方式，但现在我想要训练它以给出1或-1作为输出。当我改变 output = Sigmoid（bias + sum）; 至 output = Tanh（bias + sum）; 结果都搞砸了......

乙状结肠：

public static float Sigmoid(float x) 
{
    return 1.0f / (1.0f + (float) Mathf.Exp(-x));
}

双曲正切：

public float Tanh(float x)
{
    //return (float)(Mathf.Exp(x) - Mathf.Exp(-x)) / (Mathf.Exp(x) + Mathf.Exp(-x));
    //return (float)(1.7159f * System.Math.Tanh(2/3 * x));
    return (float)System.Math.Tanh(x);
}

你可以看到我尝试了不同的公式我发现tanh但没有输出是有意义的，我得到-1我问0或0.76159我问1或它在问及时仍然在正数和负数之间翻转 - 1和其他不匹配......

-EDIT-更新了当前正在运行的代码（将上面的calcOuput更改为我现在使用的代码）：

public float[] train(float[] inputs, float[] answer) 
{
    float[] result = feedForward(inputs);
    deltaOutput = new float[result.Length];

    for(int ii=0; ii<result.Length; ii++)
    {
        deltaOutput[ii] = 0.66666667f * (1.7159f - (result[ii]*result[ii]))  * (answer[ii]-result[ii]);
    }

    // BACKPROPOGATION

    for(int ii=0; ii<output.Length; ii++)
    {
        ArrayList connections = output[ii].getConnections();
        for (int i = 0; i < connections.Count; i++) 
        {
            Connection c = (Connection) connections[i];
            Node node = c.getFrom();
            float o = node.getOutput();
            float deltaWeight = o*deltaOutput[ii];
            c.adjustWeight(LEARNING_CONSTANT*deltaWeight);
        }
    }

    // ADJUST HIDDEN WEIGHTS
    for (int i = 0; i < hidden.Length; i++) 
    {
        ArrayList connections = hidden[i].getConnections();
        //Debug.Log(connections.Count);
        float sum  = 0;
        // Sum output delta * hidden layer connections (just one output)
        for (int j = 0; j < connections.Count; j++) 
        {
            Connection c = (Connection) connections[j];
            // Is this a connection from hidden layer to next layer (output)?
            if (c.getFrom() == hidden[i]) 
            {
                for(int k=0; k<deltaOutput.Length; k++)
                    sum += c.getWeight()*deltaOutput[k];
            }
        }    
        // Then adjust the weights coming in based:
        // Above sum * derivative of sigmoid output function for hidden neurons
        for (int j = 0; j < connections.Count; j++) 
        {
            Connection c = (Connection) connections[j];
            // Is this a connection from previous layer (input) to hidden layer?
            if (c.getTo() == hidden[i]) 
            {
                float o = hidden[i].getOutput();
                float deltaHidden = o * (1 - o);  // Derivative of sigmoid(x)
                deltaHidden *= sum;   
                Node node = c.getFrom();
                float deltaWeight = node.getOutput()*deltaHidden;
                c.adjustWeight(LEARNING_CONSTANT*deltaWeight);
            }
        } 
    }
    return  result;
}

Answer 1

我很困惑我应该改变什么。只是神经元的输出计算或反向传播的误差计算？这是输出计算：

您应该在反向传播代码中的某处使用sigmoid函数的派生。您还需要将tanh函数的导数替换为1 - (tanh(x))^2。

您的代码看起来像C＃。我明白了：

Console.WriteLine(Math.Tanh(0));     // prints 0
Console.WriteLine(Math.Tanh(-1));    // prints -0.761594155955765
Console.WriteLine(Math.Tanh(1));     // prints 0.761594155955765
Console.WriteLine(Math.Tanh(0.234)); // prints 0.229820548214317
Console.WriteLine(Math.Tanh(-4));    // prints -0.999329299739067

符合tanh情节：

我认为你读错了结果：你得到1的正确答案。您确定-1获得了tanh(0)吗？

如果您确定存在问题，请发布更多代码。

将激活功能从Sigmoid改为Tanh？

1 个答案: