Question

我目前正在尝试比较一些RNN，我只有LSTM的问题，我不明白为什么。

我正在使用相同的代码/数据集训练LSTM，SimpleRNN和GRU。对于他们所有人来说，损失正常减少。但是对于LSTM，在某一点（损失约0.4）之后，损失直接降至10e-8。如果我试图预测输出，我只有Nan。

这是代码：

nb_unit = 7
inp_shape = (maxlen, 7)
loss_ = "categorical_crossentropy"
metrics_ = "categorical_crossentropy"
optimizer_ = "Nadam"
nb_epoch = 250
batch_size = 64

model = Sequential()

model.add(LSTM( units=nb_unit, 
                input_shape=inp_shape, 
                return_sequences=True, 
                activation='softmax'))  # I just change the cell name
model.compile(loss=loss_,
              optimizer=optimizer_,
              metrics=[metrics_])

checkpoint = ModelCheckpoint("lstm_simple.h5",
                            monitor=loss_,
                            verbose=1,
                            save_best_only=True,
                            save_weights_only=False,
                            mode='auto',
                            period=1)
early = EarlyStopping( monitor='loss',
                       min_delta=0,
                       patience=10,
                       verbose=1,
                       mode='auto')

history = model.fit(X_train, y_train, 
                    validation_data=(X_test, y_test), 
                    epochs=nb_epoch, 
                    batch_size=batch_size, 
                    verbose=2, 
                    callbacks = [checkpoint, early])

这是GRU和LSTM的输出，具有相同的输入：

Input :
[[[1 0 0 0 0 0 0]
  [0 1 0 0 0 0 0]
  [0 0 0 1 0 0 0]
  [0 0 0 1 0 0 0]
  [0 1 0 0 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 1 0 0]
  [0 0 0 1 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 1 0 0]
  [0 0 0 1 0 0 0]
  [0 1 0 0 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 1 0 0]
  [0 0 0 1 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]]


LSTM predicts :
[[[ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]]]


GRU predicts :
[[[ 0.     0.54   0.     0.     0.407  0.     0.   ]
  [ 0.     0.005  0.66   0.314  0.     0.     0.001]
  [ 0.     0.001  0.032  0.957  0.     0.004  0.   ]
  [ 0.     0.628  0.     0.     0.     0.372  0.   ]
  [ 0.     0.555  0.     0.     0.     0.372  0.   ]
  [ 0.     0.     0.     0.     0.996  0.319  0.   ]
  [ 0.     0.     0.167  0.55   0.     0.     0.   ]
  [ 0.     0.486  0.     0.002  0.     0.51   0.   ]
  [ 0.     0.001  0.     0.     0.992  0.499  0.   ]
  [ 0.     0.     0.301  0.55   0.     0.     0.   ]
  [ 0.     0.396  0.001  0.007  0.     0.592  0.   ]
  [ 0.     0.689  0.     0.     0.     0.592  0.   ]
  [ 0.     0.001  0.     0.     0.997  0.592  0.   ]
  [ 0.     0.     0.37   0.55   0.     0.     0.   ]
  [ 0.     0.327  0.003  0.025  0.     0.599  0.   ]
  [ 0.     0.001  0.     0.     0.967  0.599  0.002]
  [ 0.     0.     0.     0.     0.     0.002  0.874]
  [ 0.004  0.076  0.128  0.337  0.02   0.069  0.378]
  [ 0.006  0.379  0.047  0.113  0.029  0.284  0.193]
  [ 0.006  0.469  0.001  0.037  0.13   0.295  0.193]]]

对于损失，您可以在fit（）历史的最后几行下面找到：

Epoch 116/250
Epoch 00116: categorical_crossentropy did not improve
 - 2s - loss: 0.3774 - categorical_crossentropy: 0.3774 - val_loss: 0.3945 - val_categorical_crossentropy: 0.3945

Epoch 117/250
Epoch 00117: categorical_crossentropy improved from 0.37673 to 0.08198, saving model to lstm_simple.h5
 - 2s - loss: 0.0820 - categorical_crossentropy: 0.0820 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08

Epoch 118/250
Epoch 00118: categorical_crossentropy improved from 0.08198 to 0.00000, saving model to lstm_simple.h5
 - 2s - loss: 7.5460e-08 - categorical_crossentropy: 7.5460e-08 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08

或基于时代的损失的演变。

我以前尝试过没有Softmax和MSE作为丢失功能，我没有收到任何错误。

如果需要，您可以找到笔记本和脚本以在Github上生成数据集（https://github.com/Coni63/SO/blob/master/Reber.ipynb）。

非常感谢你的支持，问候，尼古拉斯

编辑1：

根本原因似乎是Softmax功能消失了。如果我在它崩溃之前停下来并显示每个时间步的softmax总和：

LSTM :
[[ 0.112]
 [ 0.008]
 [ 0.379]
 [ 0.04 ]
 [ 0.001]
 [ 0.104]
 [ 0.021]
 [ 0.   ]
 [ 0.104]
 [ 0.343]
 [ 0.012]
 [ 0.   ]
 [ 0.23 ]
 [ 0.13 ]
 [ 0.147]
 [ 0.145]
 [ 0.152]
 [ 0.157]
 [ 0.163]
 [ 0.169]]


GRU :
[[ 0.974]
 [ 0.807]
 [ 0.719]
 [ 1.184]
 [ 0.944]
 [ 0.999]
 [ 1.426]
 [ 0.957]
 [ 0.999]
 [ 1.212]
 [ 1.52 ]
 [ 0.954]
 [ 0.42 ]
 [ 0.83 ]
 [ 0.903]
 [ 0.944]
 [ 0.976]
 [ 1.005]
 [ 1.022]
 [ 1.029]]

Softmax为0时，下一步将尝试除以0.现在我不知道如何修复它。

Answer 1

我发布我当前的解决方案，以防其他人在将来遇到此问题。

为了避免消失，我添加了一个简单的完全连接图层，其输出大小与输入相同，然后才能正常工作。该层允许LSTM / GRU / SRNN输出的另一个“配置”，并避免输出消失。

这是最终代码：

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;

namespace CapsLockChecker
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
            Application.Idle += Application_Idle;
        }

        private void Form1_Load(object sender, EventArgs e)
        {

        }

        void Application_Idle(object sender, EventArgs e)
        {
            if (Control.IsKeyLocked(Keys.CapsLock))
            {
                label1.Text = "CapsLock is On";
                pictureBox1.ImageLocation = "C:\\Users\\user\\source\\repos\\CapsLockChecker\\CapsLockChecker\\if_Circle_Green_34211.png";
            }
            else
            {
                label1.Text = "CapsLock if Off";
                pictureBox1.ImageLocation = "C:\\Users\\user\\source\\repos\\CapsLockChecker\\CapsLockChecker\\if_Circle_Red_34214.png";
            }
        }

        protected override void OnFormClosed(FormClosedEventArgs e)
        {
            Application.Idle -= Application_Idle;
            base.OnFormClosed(e);
        }
    }
}

我希望这可以帮助别人：）

Keras LSTM - 分类交叉熵降至0

编辑1：

1 个答案: