Question

我正在尝试使用Keras中的Hebbian更新来实现无监督的ANN。我在这里找到了由丹·桑德斯（Dan Saunders）制作的自定义Hebbian图层-https://github.com/djsaunde/rinns_python/blob/master/hebbian/hebbian.py （我希望在这里问有关他人代码的问题不是很差的形式）

在我在回购中使用此层的示例中，该层用作Dense / Conv层之间的中间层，但我想仅使用Hebbian层构建网络。

在此实现中，有两个关键的问题使我感到困惑：

似乎输入暗淡和输出暗淡必须相同才能使此层正常工作。为什么会这样，我该怎么做才能使它们有所不同？
为什么权重矩阵的对角线设置为零？它说这是为了“确保没有神经元横向连接到其自身”，但是我认为连接权重在上一层和当前层之间，而不是在当前层和其本身之间。

以下是Hebbian层实现的代码：

    from keras import backend as K
    from keras.engine.topology import Layer

    import numpy as np
    import tensorflow as tf

    np.set_printoptions(threshold=np.nan)

    sess = tf.Session()


    class Hebbian(Layer):


    def __init__(self, output_dim, lmbda=1.0, eta=0.0005, connectivity='random', connectivity_prob=0.25, **kwargs):
    '''
    Constructor for the Hebbian learning layer.

    args:
        output_dim - The shape of the output / activations computed by the layer.
        lambda - A floating-point valued parameter governing the strength of the Hebbian learning activation.
        eta - A floating-point valued parameter governing the Hebbian learning rate.
        connectivity - A string which determines the way in which the neurons in this layer are connected to
            the neurons in the previous layer.
    '''
    self.output_dim = output_dim
    self.lmbda = lmbda
    self.eta = eta
    self.connectivity = connectivity
    self.connectivity_prob = connectivity_prob

    if self.connectivity == 'random':
        self.B = np.random.random(self.output_dim) < self.connectivity_prob
    elif self.connectivity == 'zero':
        self.B = np.zeros(self.output_dim)

    super(Hebbian, self).__init__(**kwargs)


    def random_conn_init(self, shape, dtype=None):
    A = np.random.normal(0, 1, shape)
    A[self.B] = 0
    return tf.constant(A, dtype=tf.float32)


    def zero_init(self, shape, dtype=None):
    return np.zeros(shape)


    def build(self, input_shape):
    # create weight variable for this layer according to user-specified initialization
    if self.connectivity == 'all':
        self.kernel = self.add_weight(name='kernel', shape=(np.prod(input_shape[1:]), \
                            np.prod(self.output_dim)), initializer='uniform', trainable=False)
    elif self.connectivity == 'random':
        self.kernel = self.add_weight(name='kernel', shape=(np.prod(input_shape[1:]), \
                            np.prod(self.output_dim)), initializer=self.random_conn_init, trainable=False)
    elif self.connectivity == 'zero':
        self.kernel = self.add_weight(name='kernel', shape=(np.prod(input_shape[1:]), \
                            np.prod(self.output_dim)), initializer=self.zero_init, trainable=False)
    else:
        raise NotImplementedError

    # ensure that no neuron is laterally connected to itself
    self.kernel = self.kernel * tf.diag(tf.zeros(self.output_dim))

    # call superclass "build" function
    super(Hebbian, self).build(input_shape)


    def call(self, x):
    x_shape = tf.shape(x)
    batch_size = tf.shape(x)[0]

    # reshape to (batch_size, product of other dimensions) shape
    x = tf.reshape(x, (tf.reduce_prod(x_shape[1:]), batch_size))

    # compute activations using Hebbian-like update rule
    activations = x + self.lmbda * tf.matmul(self.kernel, x)

    # compute outer product of activations matrix with itself
    outer_product = tf.matmul(tf.expand_dims(x, 1), tf.expand_dims(x, 0))

    # update the weight matrix of this layer
    self.kernel = self.kernel + tf.multiply(self.eta, tf.reduce_mean(outer_product, axis=2))
    self.kernel = tf.multiply(self.kernel, self.B)
    self.kernel = self.kernel * tf.diag(tf.zeros(self.output_dim))

    return K.reshape(activations, x_shape)

在第一次检查时，我希望该层能够从上一层获取输入，执行简单的激活计算（输入*权重），然后根据Hebbian更新来更新权重（例如-如果激活率很高，则是b / t节点，增加权重），然后将激活信息传递到下一层。

我还希望它能够处理从一层到另一层的节点数量的减少/增加。

相反，我似乎无法弄清楚为什么输入和输出的暗点必须相同，以及为什么权重矩阵的对角线设置为零。

在代码中（隐式地或显式地）在哪里规定层必须是相同的暗淡？

在代码中（隐式或显式）的位置是该层的权重矩阵将当前层连接到其自身的规范吗？

很抱歉，如果应该将此Q分为2，但似乎它们可能与e / o有关，所以我将其保留为1。

很高兴在需要时提供更多详细信息。

编辑：意识到当我尝试创建输出暗淡与输入暗淡不同的图层时，我忘记添加我收到的错误消息：

model = Sequential()
model.add(Hebbian(input_shape = (256,1), output_dim = 256))

这会编译没有错误^

model = Sequential()
model.add(Hebbian(input_shape = (256,1), output_dim = 24))

此^引发错误： IndexError：布尔索引与维度0上的索引数组不匹配；维度为256，但相应的布尔维度为24

Answer 1

如果有人（像我这样；又是我）从Google来这里，试图在调用新输入时创建一个在线学习层，我刚刚发现了另一个问题，我认为这很重要：

Persistent Variable in keras Custom Layer

Self.call仅在定义图形时被调用，为了学习发生在每个新输入上，您需要将self.add_update添加到调用函数中。

Answer 2

好的，我想我可能已经知道了。有很多小问题，但最大的问题是我需要添加compute_output_shape函数，该函数使该层能够修改其输入的形状，如下所述： https://keras.io/layers/writing-your-own-keras-layers/

这是我所做的所有更改的代码。它将编译和修改输入形状就好了。请注意，该层计算层本身内部的权重变化，如果您尝试实际使用该层，则可能会有一些问题（我仍在解决这些问题），但这是一个单独的问题。

class Hebbian(Layer):


def __init__(self, output_dim, lmbda=1.0, eta=0.0005, connectivity='random', connectivity_prob=0.25, **kwargs):
    '''
    Constructor for the Hebbian learning layer.

    args:
        output_dim - The shape of the output / activations computed by the layer.
        lambda - A floating-point valued parameter governing the strength of the Hebbian learning activation.
        eta - A floating-point valued parameter governing the Hebbian learning rate.
        connectivity - A string which determines the way in which the neurons in this layer are connected to
            the neurons in the previous layer.
    '''
    self.output_dim = output_dim
    self.lmbda = lmbda
    self.eta = eta
    self.connectivity = connectivity
    self.connectivity_prob = connectivity_prob

    super(Hebbian, self).__init__(**kwargs)



def random_conn_init(self, shape, dtype=None):
    A = np.random.normal(0, 1, shape)
    A[self.B] = 0
    return tf.constant(A, dtype=tf.float32)


def zero_init(self, shape, dtype=None):
    return np.zeros(shape)


def build(self, input_shape):
    # create weight variable for this layer according to user-specified initialization
    if self.connectivity == 'random':
        self.B = np.random.random(input_shape[0]) < self.connectivity_prob
    elif self.connectivity == 'zero':
        self.B = np.zeros(self.output_dim)

    if self.connectivity == 'all':
        self.kernel = self.add_weight(name='kernel', shape=(np.prod(input_shape[1:]), \
                    np.prod(self.output_dim)), initializer='uniform', trainable=False)
    elif self.connectivity == 'random':
        self.kernel = self.add_weight(name='kernel', shape=(np.prod(input_shape[1:]), \
                    np.prod(self.output_dim)), initializer=self.random_conn_init, trainable=False)
    elif self.connectivity == 'zero':
        self.kernel = self.add_weight(name='kernel', shape=(np.prod(input_shape[1:]), \
                    np.prod(self.output_dim)), initializer=self.zero_init, trainable=False)
    else:
        raise NotImplementedError


    # call superclass "build" function
    super(Hebbian, self).build(input_shape)


def call(self, x):  # x is the input to the network
    x_shape = tf.shape(x)
    batch_size = tf.shape(x)[0]

    # reshape to (batch_size, product of other dimensions) shape
    x = tf.reshape(x, (tf.reduce_prod(x_shape[1:]), batch_size))

    # compute activations using Hebbian-like update rule
    activations = x + self.lmbda * tf.matmul(self.kernel, x)  


    # compute outer product of activations matrix with itself
    outer_product = tf.matmul(tf.expand_dims(x, 1), tf.expand_dims(x, 0)) 

    # update the weight matrix of this layer
    self.kernel = self.kernel + tf.multiply(self.eta, tf.reduce_mean(outer_product, axis=2)) 
    self.kernel = tf.multiply(self.kernel, self.B)
    return K.reshape(activations, x_shape)

def compute_output_shape(self, input_shape):
    return (input_shape[0], self.output_dim)

Keras中的自定义Hebbian层实现-输入/输出变暗和横向节点连接

2 个答案: