试用：

Question

因此，我在模型中使用了GloVe，并且可以使用，但是现在我改用了Elmo（引用了Keras代码在GitHub Elmo Keras Github，utils.py

上的可用信息）

但是，当我打印model.summary时，我在ELMo嵌入层中得到0个参数，这与我使用手套时不同。那是正常的吗？如果不能，请告诉我我在做什么错 使用手套我获得了超过2000万个参数

##-------->  When I was using Glove  Embedding Layer
word_embedding_layer = emb.get_keras_embedding(#dropout = emb_dropout,
                                            trainable = True,
                                            input_length = sent_maxlen, 
                                            name='word_embedding_layer') 


## --------> Deep layers
pos_embedding_layer = Embedding(output_dim =pos_tag_embedding_size, #5
                         input_dim = len(SPACY_POS_TAGS),
                         input_length = sent_maxlen, #20
                         name='pos_embedding_layer')
latent_layers = stack_latent_layers(num_of_latent_layers)



##--------> 6] Dropout 
dropout = Dropout(0.1) 

## --------> 7]Prediction
predict_layer = predict_classes()


## --------> 8] Prepare input features, and indicate how to embed them
inputs = [Input((sent_maxlen,), dtype='int32', name='word_inputs'),
            Input((sent_maxlen,), dtype='int32', name='predicate_inputs'),
            Input((sent_maxlen,), dtype='int32', name='postags_inputs')]


## --------> 9] ELMo Embedding and Concat all inputs and run on deep network
from elmo import ELMoEmbedding
import utils 

idx2word = utils.get_idx2word()
ELmoembedding1 = ELMoEmbedding(idx2word=idx2word, output_mode="elmo", trainable=True)(inputs[0]) # These two are interchangeable
ELmoembedding2 = ELMoEmbedding(idx2word=idx2word, output_mode="elmo", trainable=True)(inputs[1]) # These two are interchangeable


embeddings = [ELmoembedding1, 
               ELmoembedding2,
               pos_embedding_layer(inputs[3])]

con1 = keras.layers.concatenate(embeddings)

## --------> 10]Build model 

outputI = predict_layer(dropout(latent_layers(con1)))
model = Model(inputs, outputI)

model.compile(optimizer='adam',
                           loss='categorical_crossentropy',
                           metrics=['categorical_accuracy'])
model.summary()

试用：

注意：我尝试将TF-Hub Elmo与Keras代码一起使用，但是输出始终是2D张量[即使当我将其更改为'Elmo'设置和'LSTM'而不是默认值']，所以我无法与POS_embedding_layer串联。我尝试重塑，但最终我遇到了相同的问题，总参数0。

Answer 1

根据TF-Hub描述（https://tfhub.dev/google/elmo/2），单个单词的嵌入不可训练。仅嵌入层和LSTM层的加权总和是。因此，您应该在ELMo级别获得4个可训练的参数。

我能够使用StrongIO's example on Github中定义的类来获取可训练的参数。该示例仅提供一个输出为 default 层的类，每个输入示例（本质上是文档/句子编码器）的输出为1024。要访问每个单词的嵌入（ elmo 层），需要进行this issue中建议的一些更改：

class ElmoEmbeddingLayer(Layer):
    def __init__(self, **kwargs):
        self.dimensions = 1024
        self.trainable=True
        super(ElmoEmbeddingLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        self.elmo = hub.Module('https://tfhub.dev/google/elmo/2', trainable=self.trainable,
                               name="{}_module".format(self.name))

        self.trainable_weights += K.tf.trainable_variables(scope="^{}_module/.*".format(self.name))
        super(ElmoEmbeddingLayer, self).build(input_shape)

    def call(self, x, mask=None):
        result = self.elmo(
            K.squeeze(
                K.cast(x, tf.string), axis=1
            ),
            as_dict=True,
            signature='default',
            )['elmo']
        return result

    def compute_output_shape(self, input_shape):
        return (input_shape[0], None, self.dimensions)

您可以将ElmoEmbeddingLayer与POS层堆叠在一起。
作为更一般的示例，可以使用一维ConvNet模型中的ELMo嵌入进行分类：

elmo_input_layer = Input(shape=(None, ), dtype="string")
elmo_output_layer = ElmoEmbeddingLayer()(elmo_input_layer)

conv_layer = Conv1D(
    filters=100,
    kernel_size=3,
    padding='valid',
    activation='relu',
    strides=1)(elmo_output_layer)

pool_layer = GlobalMaxPooling1D()(conv_layer)
dense_layer = Dense(32)(pool_layer)
output_layer = Dense(1, activation='sigmoid')(dense_layer)

model = Model(
    inputs=elmo_input_layer,
    outputs=output_layer)
model.summary()

模型摘要如下：

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_62 (InputLayer)        (None, None)              0         
_________________________________________________________________
elmo_embedding_layer_13 (Elm (None, None, 1024)        4         
_________________________________________________________________
conv1d_46 (Conv1D)           (None, None, 100)         307300    
_________________________________________________________________
global_max_pooling1d_42 (Glo (None, 100)               0         
_________________________________________________________________
dense_53 (Dense)             (None, 32)                3232      
_________________________________________________________________
dense_54 (Dense)             (None, 1)                 33        
=================================================================
Total params: 310,569
Trainable params: 310,569
Non-trainable params: 0
_________________________________________________________________

在Keras Elmo嵌入层中有0个参数吗？这正常吗？

试用：

1 个答案: