Question

尝试deal with missing data，我编写了以下模型并运行它。输出如下。为什么对NaN期望值进行训练，而NaN期望值却被loss_0_where_nan掩盖了（历史表明损失确实被评估为0.0），但是在权重矩阵中引入了NaN权重hidden和max_min_pred中的哪一个？我首先认为这可能是对单个参数学习和输出值的某种加权，我认为这可能是Adadelta优化器所特有的。但是对于SGD也是如此。

import keras
from keras.models import Model
from keras.optimizers import Adadelta
from keras.losses import mean_squared_error
from keras.layers import Input, Dense

import tensorflow as tf
import numpy

def loss_0_where_nan(loss_function, msg=""):
    def filtered_loss_function(y_true, y_pred):
        with_nans = loss_function(y_true, y_pred)
        nans = tf.is_nan(with_nans)
        filtered = tf.where(nans, tf.zeros_like(with_nans), with_nans)
        filtered = tf.Print(filtered,
                            [y_true, y_pred, nans, with_nans, filtered],
                            message=msg)
        return filtered
    return filtered_loss_function

input = Input(shape=(3,))

hidden = Dense(2)(input)
min_pred = Dense(1)(hidden)
max_min_pred = Dense(1)(hidden)

model = Model(inputs=[input],
              outputs=[min_pred, max_min_pred])

model.compile(
    optimizer=Adadelta(),
    loss=[loss_0_where_nan(mean_squared_error, "aux: "),
          loss_0_where_nan(mean_squared_error, "main: ")],
    loss_weights=[0.2, 1.0])

def random_values(n, missing=False):
    for i in range(n):
        x = numpy.random.random(size=(2, 3))
        _min = numpy.minimum(x[..., 0], x[..., 1])
        if missing:
            _max_min = numpy.full((len(x), 1), numpy.nan)
        else:
            _max_min = numpy.maximum(_min, x[..., 2]).reshape((-1, 1))
        # print(x, numpy.array(_min).reshape((-1, 1)), numpy.array(_max_min), sep="\n", end="\n\n")
        yield x, [numpy.array(_min).reshape((-1, 1)), numpy.array(_max_min)]

model.fit_generator(random_values(2, False),
                    steps_per_epoch=2,
                    verbose=False)
print("With missing")
history = model.fit_generator(random_values(1, True),
                              steps_per_epoch=1,
                    verbose=False)
print("Normal")
model.fit_generator(random_values(2, False),
                    steps_per_epoch=2,
                    verbose=False)

print(history.history)

输出：

main: [[0.29131493][0.769406676]][[-1.38235903][-3.32388687]][0 0][2.80118465 16.7550526][2.80118465 16.7550526]
aux: [[0.0422333851][0.0949674547]][[1.01466811][0.648737907]][0 0][0.945629239 0.306661695][0.945629239 0.306661695]
main: [[0.451149166][0.671600938]][[-2.46504498][-2.74316335]][0 0][8.50418854 11.6606159][8.50418854 11.6606159]
aux: [[0.451149166][0.355992794]][[0.893445313][0.917516708]][0 0][0.195625886 0.315309107][0.195625886 0.315309107]
With missing
aux: [[0.406784][0.44401589]][[0.852455556][1.23527527]][0 0][0.198623136 0.62609148][0.198623136 0.62609148]
main: [[nan][nan]][[-3.2140317][-2.22139478]][1 1][nan nan][0 0]
Normal
aux: [[0.490041673][0.00489727268]][[nan][nan]][1 1][nan nan][0 0]
main: [[0.867286][0.949406743]][[nan][nan]][1 1][nan nan][0 0]
aux: [[0.630184174][0.391073674]][[nan][nan]][1 1][nan nan][0 0]
main: [[0.630184174][0.391073674]][[nan][nan]][1 1][nan nan][0 0]
{'loss': [0.08247146010398865], 'dense_1_loss': [0.41235730051994324], 'dense_2_loss': [0.0]}

Answer 1

似乎与tf.where()的{{3}}类似。

当y_true为nan时，filtered = tf.where(nans, tf.zeros_like(with_nans), with_nans)的斜率的计算类似于d/dw (filtered) = 1 * d/dw (tf.zeros_like) + 0 * d/dw (with_nans)。由于在这种情况下，d/dw (with_nans)是nan，所以最终的梯度是1 * 0 + 0 * nan = nan。

为避免此问题，您可以将nan设置为0，而不是将y_true的损失值设置为y_pred，以便每次{{ 1}}是y_true。

nan

由于def filtered_loss_function(y_true, y_pred): nans = tf.is_nan(y_true) masked_y_true = tf.where(nans, y_pred, y_true) filtered = loss_function(masked_y_true, y_pred) return filtered不再取决于filtered的值（在进入损失函数之前这些值被屏蔽了），因此渐变将没有nan s。

nan

期望值中的NaN，即使被屏蔽，也会在权重矩阵中引入NaN

1 个答案: