Question

我正在尝试使用Keras训练和构建令牌生成器，这是我正在执行此操作的代码段：

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense

txt1="""What makes this problem difficult is that the sequences can vary in length,
be comprised of a very large vocabulary of input symbols and may require the model 
to learn the long term context or dependencies between symbols in the input sequence."""

#txt1 is used for fitting 
tk = Tokenizer(nb_words=2000, lower=True, split=" ",char_level=False)
tk.fit_on_texts(txt1)

#convert text to sequencech
t= tk.texts_to_sequences(txt1)

#padding to feed the sequence to keras model
t=pad_sequences(t, maxlen=10)

在测试Tokenizer学习了哪些单词后，得出的结论是它仅学习了字符而没有单词。

print(tk.word_index)

输出：

{'e': 1, 't': 2, 'n': 3, 'a': 4, 's': 5, 'o': 6, 'i': 7, 'r': 8, 'l': 9, 'h': 10, 'm': 11, 'c': 12, 'u': 13, 'b': 14, 'd': 15, 'y': 16, 'p': 17, 'f': 18, 'q': 19, 'v': 20, 'g': 21, 'w': 22, 'k': 23, 'x': 24}

为什么没有任何单词？

此外，如果我打印t，它清楚地表明，单词被忽略，并且每个单词都由char逐个字符化

print(t)

输出：

[[ 0  0  0 ...  0  0 22]
 [ 0  0  0 ...  0  0 10]
 [ 0  0  0 ...  0  0  4]
 ...
 [ 0  0  0 ...  0  0 12]
 [ 0  0  0 ...  0  0  1]
 [ 0  0  0 ...  0  0  0]]

Answer 1

尝试

from keras.preprocessing.text import Tokenizer
txt1='What makes this problem difficult is that the sequences can vary in length,
be comprised of a very large vocabulary of input symbols and may require the model 
to learn the long term context or dependencies between symbols in the input sequence.'

t = Tokenizer()
t.fit_on_texts(txt1)
# summarize what was learned
print(t.word_counts)
print(t.document_count)
print(t.word_index)
print(t.word_docs)

复制并粘贴并运行。我假设问题首先出在输入文本“您有3个引号”周围的引号中。其次，您不必执行t= tk.texts_to_sequences(txt1) 而是这样做

encoded_txt = t.texts_to_matrix(txt1, mode='count')
print(encoded_txt)

其他解决方法是

from keras.preprocessing.text import text_to_word_sequence
text = txt1
# estimate the size of the vocabulary
words = set(text_to_word_sequence(text))
vocab_size = len(words)
print(vocab_size)

Answer 2

我发现了错误。如果文本是通过以下方式传递的：

txt1=["""What makes this problem difficult is that the sequences can vary in length,
be comprised of a very large vocabulary of input symbols and may require the model 
to learn the long term context or dependencies between symbols in the input sequence."""]

使用方括号，它将可以正常工作。这是t的新输出：

print(t)


[[30 31 32 33 34  5  2  1  4 35]]

这意味着该函数接受一个列表，而不只是一个文本

在标记Keras时如何忽略字符

2 个答案: