Question

使用tensorflow库中的AUC时遇到问题。我每批训练我的模型（卷积神经网络）（我不使用验证集），并且在每个时期之后我使用独立的测试集来获得我的评估。问题在于AUC评估。

在每个批次中，我计算训练集的AUC /准确度/损失/精确度/召回/ F1_score，然后我汇总这些分数的平均值。当我尝试为测试集做同样的事情时，我再次计算相同的分数。我注意到除了AUC之外的所有分数都有不同的值。我认为测试的损失功能不正确，AUC也会增加。问题是测试的AUC几乎与训练的AUC相同（即使它们的准确性，丢失误差完全不同）。

    with tf.name_scope("output"):
        W = tf.Variable(tf.truncated_normal([num_filters_total, num_classes], stddev=0.1), name="W")
        b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name="b")
        scores = tf.nn.xw_plus_b(h_drop, W, b, name="scores")
        predictions = tf.argmax(scores, 1, name="predictions")
        l2_loss += tf.nn.l2_loss(W, name="l2_loss")
        l2_loss += tf.nn.l2_loss(b, name="l2_loss")
        tf.summary.histogram("l2", l2_loss)
        tf.summary.histogram("weigths", W)
        tf.summary.histogram("biases", b)

    with tf.name_scope("auc_score"):
        # labelOut = tf.argmax(y_place_holder, 1)
        probability = tf.nn.softmax(scores)
        # auc_scoreTemp = streaming_auc(y_place_holder, probability, curve="PR")
        auc_scoreTemp = tf.metrics.auc(y_place_holder, probability, curve="PR")
        auc_score = tf.reduce_mean(tf.cast(auc_scoreTemp, tf.float32), name="auc_score")
        tf.summary.scalar("auc_score", auc_score)

    with tf.name_scope("accuracy"):
        labelOut = tf.argmax(y_place_holder, 1)
        correct_prediction = tf.equal(predictions, tf.argmax(y_place_holder, 1), name="correct_prediction")
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name="accuracy")
        tf.summary.scalar("accuracy", accuracy)


    sess.run(tf.global_variables_initializer())
    sess.run(tf.local_variables_initializer())
    for batch in batches:
        x_batch, y_batch = list(zip(*batch))
        _,  accuracy_train, auc_training,  loss_train,  prec_batch, recall_batch, f1_batch \
            = sess.run([train_step,  accuracy, auc_score,  cross_entropy,  precision_mini_batch,
                        recall_mini_batch, f1_score_min_batch], feed_dict={x_place_holder: x_batch,
                                                                           y_place_holder: y_batch,
                                                                           emb_place_holder: vocab_inv_emb_dset,
                                                                           dropout_keep_prob: dropout_rate})
    ...
       for test_batch in test_batches:
                auc_test = None
                x_test_batch, y_test_batch = list(zip(*test_batch))

                accuracy_test, loss_test,  auc_test = sess.run([accuracy, cross_entropy, auc_score],
                       feed_dict={x_place_holder: x_test_batch,
                       y_place_holder: y_test_batch,
                       emb_place_holder: vocab_inv_emb_dset_val,
                       dropout_keep_prob: 1.0})

我还尝试使用始终返回1的streaming_auc。

修改

在每个纪元的最后，我通过运行：

重置局部变量

sess.run(tf.local_variables_initializer())

但是第一批输出结果非常糟糕。在第一批之后，我从测试集中获得了与训练结果不相近的正常结果。我不知道这是否是正确的做法，但结果似乎更加现实。

Answer 1

所有tf.metrics都会返回值和更新操作（请参阅here）。因此，如所述here所述，您希望使用更新操作来累积值，然后评估auc_score以检索累积值，如下所示：

...
auc_score, auc_op = tf.metrics.auc(y_place_holder, probability, curve="PR")
...
for batch in batches:
  sess.run([train_step,  accuracy, auc_op,  cross_entropy,...)
...
py_auc = sess.run(auc)

编辑 - 显示tf.metrics.auc和tf.contrib.metrics.streaming_auc

的玩具示例

import tensorflow as tf
from tensorflow.contrib import metrics
batch_sz = 100
noise_mag = 0.5
nloop = 10

tf.set_random_seed(0)

batch_x = tf.random_uniform([batch_sz, 1], 0, 2, dtype=tf.int32)
noise = noise_mag * tf.random_normal([batch_sz, 1])
batch_y = tf.sigmoid(tf.to_float(batch_x) + noise)

auc_val, auc_accum = tf.metrics.auc(batch_x, batch_y)

#note: contrib.metrics.streaming_auc reverses labels, predictions
auc_val2, auc_accum2 = metrics.streaming_auc(batch_y, batch_x)   

with tf.Session() as sess:
    sess.run(tf.local_variables_initializer())

    for i in range(nloop):
        _ = sess.run([auc_accum, auc_accum2])

    auc, auc2 = sess.run([auc_val, auc_val2])
    print('Accumulated AUC = ', sess.run(auc_val))  #0.9238014
    print('Accumulated AUC2 = ', sess.run(auc_val)) #0.9238014

Tensorflow报告错误的AUC

1 个答案: