为什么在同一数据上训练准确度为99％，但预测准确度为81％？

Question

我查找了与此问题类似的问题，但我仍然不明白为什么会给出这样的结果。模型训练高达99％的精度是正常的，但是用于预测相同的精确数据时，它给出的精度较低，在这种情况下为81％？它不应该返回99％的准确性吗？

此外，当我呈现新的看不见的数据时，预测准确性高达17％。当然这是不对的。我了解该模型在提供新数据时应低于模型的准确性，但绝不至于达到17％。

这是上下文的代码。我放置评论以便于阅读：

# Step 1) Split Data into Training and Prediction Sets
num_split_df_at = int(0.75*len(df))
np_train_data = df.iloc[0:num_split_df_at, columns_index_list].to_numpy()
np_train_target = list(df.iloc[0:num_split_df_at, 4])
np_predict_data = df.iloc[num_split_df_at:len(df), columns_index_list].to_numpy()
np_predict_target = list(df.iloc[num_split_df_at:len(df), 4])

# Step 2) Split Training Data into Training and Validation Sets
x_train, x_test, y_train, y_test = train_test_split(np_train_data, np_train_target, random_state=0)


# Step 3) Reshape Training and Validation Sets to (49, 5)
# prints: "(3809, 245)"
print(x_train.shape)
# prints: "(1270, 245)"
print(x_test.shape)
x_train = x_train.reshape(x_train.shape[0], round(x_train.shape[1]/5), 5)
x_test = x_test.reshape(x_test.shape[0], round(x_test.shape[1]/5), 5)
y_train = np.array(y_train)- 1
y_test = np.array(y_test)- 1
# prints: "(3809, 49, 5)"
print(x_train.shape)
# prints: "[0 1 2 3 4 5 6 7 8 9]"
print(np.unique(y_train))
# prints: "10"
print(len(np.unique(y_train)))

input_shape = (x_train.shape[1], 5)

# Step 4) Run Model
adam = keras.optimizers.Adam(learning_rate=0.0001)
model = Sequential()
model.add(Conv1D(512, 5, activation='relu', input_shape=input_shape))
model.add(Conv1D(512, 5, activation='relu'))
model.add(MaxPooling1D(3))
model.add(Conv1D(512, 5, activation='relu'))
model.add(Conv1D(512, 5, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
    
model.fit(x_train, y_train, batch_size=128, epochs=150, validation_data=(x_test, y_test))
print(model.summary())
model.save('model_1')

# Step 5) Predict on Exact Same Trained Data - Should Return High Accuracy
np_train_data = np_train_data.reshape(np_train_data.shape[0], round(np_train_data.shape[1]/5), 5)
np_train_target = np.array(np_train_target)- 1
predict_results = model.predict_classes(np_train_data)
print(accuracy_score(predict_results, np_train_target))

# Step 6) Predict on Validation Set
np_predict_data = np_predict_data.reshape(np_predict_data.shape[0], round(np_predict_data.shape[1]/5), 5)
np_predict_target = np.array(np_predict_target)- 1
predict_results = model.predict_classes(np_predict_data)
print(accuracy_score(predict_results, np_predict_target))

以下是预测结果：

我的输入数据与此类似-49天，每天5个数据点：

我输出的可能的分类结果是：

[1 2 3 4 5 6 7 8 9 10] 转换为 [0 1 2 3 4 5 6 7 8 9] < / strong> 表示“ sparse_categorical_crossentropy”

Answer 1

这是因为Keras模型的训练准确性/损失是分批计算，然后取平均值（see here）。取而代之的是，对所有传递的数据同时计算验证指标/性能。

这只是在此虚拟示例中进行验证。我们训练一个NN，并将相同的训练数据作为有效数据传递。通过这种方式，我们可以比较（a）训练acc，（b）验证acc和（c）训练结束时的precision_score。我们可以看到（b）=（c），但由于上述原因，（a）与（c）和（b）不同

timestamp, features, n_sample = 45, 2, 1000
n_class = 10
X = np.random.uniform(0,1, (n_sample, timestamp, features))
y = np.random.randint(0,n_class, n_sample)

model = Sequential()
model.add(Conv1D(8, 3, activation='relu', input_shape=(timestamp, features)))
model.add(MaxPooling1D(3))
model.add(Conv1D(8, 3, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(n_class, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
history = model.fit(X, y, batch_size=128, epochs=5, validation_data=(X, y))

history.history['accuracy'][-1] # (a)
history.history['val_accuracy'][-1] # (b)
accuracy_score(y, np.argmax(model.predict(X), axis=1)) # (c)