将生成器与tf.py_function一起使用

时间:2020-04-11 13:53:30

标签: tensorflow machine-learning

如何将生成器函数传递给tf.py_function?

def generate_context_target_pairs(words, window_size, vocab_size):
  context_length = window_size*2
  sentence_length = len(words)
  for index, word in enumerate(words):
      context_words = []
      label_word   = []            
      start = index - window_size
      end = index + window_size + 1

      context_words.append([words[i] 
                            for i in range(start, end) 
                            if 0 <= i < sentence_length 
                            and i != index])
      label_word.append(word)

      x = sequence.pad_sequences(context_words, maxlen=context_length,
                                  padding = 'post', truncating = 'post',value = encoder.encode("<UNK>"))
      y = to_categorical(label_word, vocab_size) #creates a one hot vector for each sequence in the list
      return (tf.convert_to_tensor(x, dtype=tf.float32), tf.convert_to_tensor(y, dtype=tf.float32))

def generate_context_target_pairs_map_fxn(word_list, window_size, vocab_size):
  # py_func doesn't set the shape of the returned tensors.
  encoded_text, label = tf.py_function(generate_context_target_pairs, 
                                       inp=[word_list, window_size, vocab_size], 
                                       Tout=(tf.float32, tf.float32))
  return encoded_text, label

如果在上述函数中我想传递一个生成器而不是python函数,是否可能?,如果可以,请分享一些示例。这真的很有帮助,我查看了官方文档,但找不到任何具体内容。

0 个答案:

没有答案