Question

我正在使用Keras构建一个简单的神经网络。它具有活动正则化，因此唯一隐藏层的输出被强制具有较小的值。这是代码：

import numpy as np
import math
import keras
from keras.models import Model, Sequential
from keras.layers import Input, Dense, Activation
from keras import regularizers
from keras import backend as K

a=1

def my_regularizer(inputs):
     means=K.mean((inputs),axis=1)
     return a*K.sum(means)**2

x_train=np.random.uniform(low=-1,high=1,size=(200,2))

model=Sequential([
     Dense(20,input_shape=(2,),activity_regularizer=my_regularizer),
     Activation('tanh'),
     Dense(2,),
     Activation('linear')
])

model.compile(optimizer='adam',loss='mean_squared_error')
model.fit(x_train,x_train,epochs=20,validation_split=0.1)

问题：

1）当前，参数 a 设置在开头，并且不会更改。如何更改代码，以便每次迭代后更新参数 a ，使得

a_new = f（a_old，input）

其中input是隐藏层的值，而 f（。）是任意函数。

2）我希望在应用第一个激活函数tanh之后应用我的活动正则化函数。我是否正确编写了代码？

中的术语“ activity_regularizer = my_regularizer”

Dense(20,input_sahpe=(2,),activity_regularizer=my_regularizer)

使我感到正则化函数已应用于激活函数tanh之前的值。

Answer 1

您可以-但首先，您需要一个有效的Keras Regularizer对象（您的函数将不起作用）：

class MyActivityRegularizer(Regularizer):
    def __init__(self, a=1):
        self.a = K.variable(a, name='a')

    # gets called at each train iteration
    def __call__(self, x): # your custom function here
        means = K.mean(x, axis=1)
        return self.a * K.sum(means)**2

    def get_config(self): # required class method
        return {"a": float(K.get_value(self.a))}

接下来，要使用.fit，您需要一个自定义Keras Callback对象（请参阅底部的替代方法）：

class ActivityRegularizerScheduler(Callback):
    """ 'on_batch_end' gets automatically called by .fit when finishing
    iterating over a batch. The model, and its attributes, are inherited by 
    'Callback' (except at __init__) and can be accessed via, e.g., self.model """

    def __init__(self, model, update_fn):
        self.update_fn=update_fn
        self.activity_regularizers=_get_activity_regularizers(model)

    def on_batch_end(self, batch, logs=None):
        iteration = K.get_value(self.model.optimizer.iterations)
        new_activity_reg = self.update_fn(iteration) 

        # 'activity_regularizer' references model layer's activity_regularizer (in this 
        # case 'MyActivityRegularizer'), so its attributes ('a') can be set directly
        for activity_regularizer in self.activity_regularizers:
            K.set_value(activity_regularizer.a, new_activity_reg)

def _get_activity_regularizers(model):
    activity_regularizers = []
    for layer in model.layers:
        a_reg = getattr(layer,'activity_regularizer',None)
        if a_reg is not None:
            activity_regularizers.append(a_reg)
    return activity_regularizers

最后，您需要在Keras CustomObjectScope内创建模型-全文请参见。下面。

用法示例：

from keras.layers import Dense
from keras.models import Sequential
from keras.regularizers import Regularizer
from keras.callbacks import Callback
from keras.utils import CustomObjectScope
from keras.optimizers import Adam
import keras.backend as K
import numpy as np

def make_model(my_reg):
    return Sequential([
        Dense(20, activation='tanh', input_shape=(2,), activity_regularizer=my_reg),
        Dense(2,  activation='linear'),
        ])

my_reg = MyActivityRegularizer(a=1)

with CustomObjectScope({'MyActivityRegularizer':my_reg}): # required for Keras to recognize
    model = make_model(my_reg)
opt = Adam(lr=1e-4)
model.compile(optimizer=opt, loss='mse')

x = np.random.randn(320,2) # dummy data
y = np.random.randn(320,2) # dummy labels

update_fn = lambda x: .5 + .4*np.cos(x) #x = number of train updates (optimizer.iterations)
activity_regularizer_scheduler = ActivityRegularizerScheduler(model, update_fn)

model.fit(x,y,batch_size=32,callbacks=[activity_regularizer_scheduler],
          epochs=4,verbose=1)

要跟踪您的a并确保其发生变化，您可以通过以下方式在每个纪元结束时获取其值：

for epoch in range(4):
    model.fit(x,y,batch_size=32,callbacks=[activity_regularizer_scheduler],epochs=1)
    print("Epoch {} activity_regularizer 'a': {}".format(epoch,
            K.get_value(_get_activity_regularizers(model)[0].a)))
# My output:
# Epoch 0 activity_regularizer 'a': 0.7190816402435303
# Epoch 1 activity_regularizer 'a': 0.4982417821884155
# Epoch 2 activity_regularizer 'a': 0.2838689386844635
# Epoch 3 activity_regularizer 'a': 0.8644570708274841

关于（2），恐怕您是对的-'tanh'输出将不会使用；您需要改为传递activation='tanh'。

最后，您可以通过train_on_batch 进行回调，而无需回调-但缺点是，您需要自己向模型中馈送数据（并对其进行混洗等）： >

activity_regularizers = _get_activity_regularizers(model)

for iteration in range(100):
   x, y = get_data()
   model.train_on_batch(x,y)
   iteration = K.get_value(model.optimizer.iterations)

   for activity_regularizer in activity_regularizers:
       K.set_value(activity_regularizer, update_fn(iteration))

Keras具有activity_regularizer，每次迭代都会更新

1 个答案: