Question

我正在使用LSTM和从通用语句编码器获得的嵌入来解决多类分类问题。

以前，我使用的是Glove嵌入，并且得到了LSTM所需的输入形状（batch_size，timesteps，input_dim）。我打算使用通用句子编码器，发现通用句子编码器的输出是2d [批处理，功能]。如何进行所需的更改。

LSTM +通用句子编码器

EMBED_SIZE = 512

module_url = "https://tfhub.dev/google/universal-sentence-encoder-large/3"
embed = hub.Module(module_url)

def UniversalEmbedding(x):
    return embed(tf.squeeze(tf.cast(x, tf.string)), 
            signature="default", as_dict=True)["default"]


seq_input = Input(shape=(MAX_SEQUENCE_LENGTH,),dtype='int32')
print("seq i",seq_input.shape,seq_input)

embedded_seq = Lambda(UniversalEmbedding,                          
                          output_shape=(EMBED_SIZE,))(seq_input)
print("EMD SEQ",embedding.shape,type(embedded_seq))

# (timesteps, n_features) (,MAX_SEQUENCE_LENGTH, EMBED_SIZE) (,150,512)
x_1 = LSTM(units=NUM_LSTM_UNITS,
                            name='blstm_1',
                        dropout=DROP_RATE_LSTM)(embedded_seq)
print(x_1)

这会产生以下错误

seq i (?, 150) Tensor("input_8:0", shape=(?, 150), dtype=int32)
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
I0529 07:24:32.504808 140127577749376 saver.py:1483] Saver not created because there are no variables in the graph to restore
EMD SEQ (?, 512) <class 'tensorflow.python.framework.ops.Tensor'>
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-34-ea634319205b> in <module>()
     12 x_1 = LSTM(units=NUM_LSTM_UNITS,
     13                             name='blstm_1',
---> 14                         dropout=DROP_RATE_LSTM)(embedded_seq)
     15 print(x_1)
     16 

2 frames
/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py in assert_input_compatibility(self, inputs)
    309                                      self.name + ': expected ndim=' +
    310                                      str(spec.ndim) + ', found ndim=' +
--> 311                                      str(K.ndim(x)))
    312             if spec.max_ndim is not None:
    313                 ndim = K.ndim(x)

ValueError: Input 0 is incompatible with layer blstm_1: expected ndim=3, found ndim=2

LSTM +手套嵌入

embedding_layer = Embedding(nb_words,
                            EMBED_SIZE, 
                            weights=[embedding_matrix],
                            input_length=MAX_SEQUENCE_LENGTH,
                            trainable=False)

seq_input = Input(shape=(MAX_SEQUENCE_LENGTH,),dtype='int32')
print("SEQ INP",seq_input,seq_input.shape)
embedded_seq = embedding_layer(seq_input)
print("EMD SEQ",embedded_seq.shape)

# Bi-directional LSTM  # (timesteps, n_features)
x_1 = Bidirectional(LSTM(units=NUM_LSTM_UNITS,
                         name='blstm_1',
                         dropout=DROP_RATE_LSTM,
                         recurrent_dropout=DROP_RATE_LSTM),
                    merge_mode='concat')(embedded_seq)
x_1 = Dropout(DROP_RATE_DENSE)(x_1)
x_1 = Dense(NUM_DENSE_UNITS,activation='relu')(x_1)
x_1 = Dropout(DROP_RATE_DENSE)(x_1)

输出（这在LSTM上正常工作）

SEQ INP Tensor("input_2:0", shape=(?, 150), dtype=int32) (?, 150)
EMD SEQ (?, 150, 300)

Answer 1

Sentence Encoder与word2vec或Glove不同，它不是词级嵌入：

模型经过训练和优化，适用于长度超过单词的文本，例如句子，词组或简短的段落。它是在各种数据源和各种任务，旨在动态适应各种自然语言了解任务。输入的是可变长度的英文文本，输出是512维向量。我们将此模型应用于STS 语义相似性的基准，结果可以在提供示例笔记本。通用句子编码器模型使用深度平均网络（DAN）编码器进行培训。

在上面的示例中，他们使用“ lambda”函数用于FF神经网络，而下一层的输入是2D，这与CNN（3D）的RNN不同。

很快，您需要做的是准备文本，然后再通过嵌入层将其输入网络：

def process_text(sentences_list):
    path = './processed_data'
    embeddings_file = "embeddings-{}.pickle".format(len(sentences_list))
    if not os.path.isfile(join(path, embeddings_file)):
        module_url = "https://tfhub.dev/google/universal-sentence-encoder-large/3"
        embed = hub.Module(module_url)
        with tf.Session() as sess:
            sess.run([tf.global_variables_initializer(), tf.tables_initializer()])
            sentences_list = sess.run(embed(sentences_list))
        sentences_list = np.array(sentences_list)
        sentences_list = np.array([np.reshape(embedding, (len(embedding), 1)) for embedding in sentences_list])
        pickle.dump(sentences_list, open(embeddings_file, 'wb'))
    else:
        sentences_list = pickle.load(open(join(path, embeddings_file), 'rb'))
    return sentences_list

建议像我在示例中一样保存生成的嵌入内容，因为检索嵌入内容会花费一些时间。

来源：Sentiment Analysis on Twitter Data using Universal Sentence Encoder

从通用语句编码器输出生成LSTM输入

1 个答案: