Question

我正在尝试以这样的方式来连接两层：在连接时为各层分配可训练的权重。其背后的想法是，我的模型可以确定在连接时应该给哪一层更高的权重。

我已阅读此代码[https://stackoverflow.com/a/62595957/12848819][1]

class WeightedAverage(Layer):

def __init__(self, n_output):
    super(WeightedAverage, self).__init__()
    self.W = tf.Variable(initial_value=tf.random.uniform(shape=[1,1,n_output], minval=0, maxval=1),
        trainable=True) # (1,1,n_inputs)

def call(self, inputs):

    # inputs is a list of tensor of shape [(n_batch, n_feat), ..., (n_batch, n_feat)]
    # expand last dim of each input passed [(n_batch, n_feat, 1), ..., (n_batch, n_feat, 1)]
    inputs = [tf.expand_dims(i, -1) for i in inputs]
    inputs = Concatenate(axis=-1)(inputs) # (n_batch, n_feat, n_inputs)
    weights = tf.nn.softmax(self.W, axis=-1) # (1,1,n_inputs)
    # weights sum up to one on last dim

    return tf.reduce_sum(weights*inputs, axis=-1) # (n_batch, n_feat)

但是这一步执行各层的加权平均。请帮忙。如果您还有其他问题，请告诉我。谢谢。

Answer 1

我使用了加权总和（不是平均值）

class WeightedSum(layers.Layer):
    """A custom keras layer to learn a weighted sum of tensors"""

    def __init__(self, **kwargs):
        super(WeightedSum, self).__init__(**kwargs)

    def build(self, input_shape=1):
        self.a = self.add_weight(name='alpha',
                                 shape=(1),
                                 initializer=tf.keras.initializers.Constant(0.5),
                                 dtype='float32',
                                 trainable=True,
                                 constraint=tf.keras.constraints.min_max_norm(
                                     max_value=1, min_value=0))
        super(WeightedSum, self).build(input_shape)

    def call(self, model_outputs):
        return self.a * model_outputs[0] + (1 - self.a) * model_outputs[1]

    def compute_output_shape(self, input_shape):
        return input_shape[0]

Answer 2

您可以执行功能模型：

W = tf.Variable(1 ,trainable=True )
input1 = tf.keras.Input(shape=(32,))
input2 = tf.keras.Input(shape=(32,))

x1 = tf.keras.layers.Dense(8)(input1)
x2 = tf.keras.layers.Dense(8)(input2)

#merge two layers (x1 x2)and add weight to a layer of them
concatted = tf.keras.layers.Concatenate()([ w * x1 , x2 ])

在连接图层时如何添加可训练的权重

2 个答案: