Question

对于Uni的一个项目，我正在使用TensorFlow中的神经网络来实现问题回答（bAbI数据集任务5，参见https://research.fb.com/downloads/babi/）系统，我想使用TFRecords我的输入管道。

我的想法是TFRecords术语中的一个例子应该包括问题的上下文，问题本身，答案和支持句号（int指向上下文中最重要的句子以便能够回答问题）。以下是我定义函数的方法：

def make_example(context, question, answer, support):
 ex = tf.train.SequenceExample()

 fl_context = ex.feature_lists.feature_list["context"]
 fl_question = ex.feature_lists.feature_list["question"]
 fl_answer = ex.feature_lists.feature_list["answer"]
 ex.context.feature["support"].int64_list.value.append(support)

 for token in context:
    fl_context.feature.add().int64_list.value.append(token)
 for qWord in question:
    fl_question.feature.add().int64_list.value.append(qWord)
 for ansWord in answer:
    fl_answer.feature.add().int64_list.value.append(ansWord)
 fl_support.feature.add().int64_list.value.append(support)   

return ex

然而，在传递上下文，问题和答案之前，我想嵌入单词并用GloVe向量表示它们，即用（m，d）矩阵表示，其中m是句子中的标记数， d是每个单词矢量具有的维数。我得到的make_example函数似乎无法很好地处理这个问题：

theTypeError: (array([[ -9.58490000e-01,   1.73210000e-01,   
2.51650000e-01,
 -5.61450000e-01,  -1.21440000e-01,   1.54350000e+00,
 -1.28930000e+00,  -9.77790000e-01,  -1.35480000e-01,
 -6.06930000e-01,  -1.37810000e+00,   6.33470000e-01,
  1.33160000e-01,   2.46320000e-01,   6.60260000e-01,
 -4.46130000e-02,   4.09510000e-01,  -7.61670000e-01,
  4.67530000e-01,  -6.67810000e-01,   2.99850000e-01,
 -2.74810000e-01,  -5.47990000e-01,  -8.56820000e-01,
  5.30880000e-02,  -2.01700000e+00,   7.48530000e-01,
 -1.27830000e-01,   1.32050000e-01,  -2.19450000e-01,
  2.29830000e+00,  -3.17680000e-01,  -8.64940000e-01,
 -1.08630000e-01,  -8.13770000e-02,  -7.03420000e-01,
  4.60000000e-01,  -3.34730000e-01,   4.37030000e-02,
 -7.55080000e-01,  -6.89710000e-01,   7.14380000e-01,
 -8.35950000e-02,   1.58620000e-02,  -5.23850000e-01,
  1.72520000e-01,  -4.98740000e-01,   2.30810000e-01,
 -3.64690000e-01,   1.5 has type <class 'tuple'>, but expected one of: 
(<class 'int'>,)

指向上面的fl_context.feature.add().int64_list.value.append(token) ...有人能指出我误解了TFRecords的概念，并给我一个如何解决问题的建议吗？
我经常搜索学习材料，但通常TFRecords上的例子都是图像数据。到目前为止，我的推荐信是https://medium.com/@TalPerry/getting-text-into-tensorflow-with-the-dataset-api-ffb832c8bec6和http://web.stanford.edu/class/cs20si/lectures/notes_09.pdf。

提前多多感谢！

Answer 1

我的问题的解决方案可以在这里找到：https://github.com/simonada/q-and-a-tensorflow/blob/master/src/Q%26A%20with%20TF-%20TFRecords%20and%20Eager%20Execution.ipynb

我的方法如下：

将文本存储到csv文件中：每行（上下文，问题，答案）

在我的情况下，定义将序列转换为tf_example的函数

def sequence_to_tf_example(context, question, answer):
    context_ids= vectorize(context, False, word_to_index)
    question_ids= vectorize(question, False, word_to_index)
    answer_ids= vectorize(answer, True, word_to_index)
    ex = tf.train.SequenceExample()

    context_tokens = ex.feature_lists.feature_list["context"]
    question_tokens = ex.feature_lists.feature_list["question"]
    answer_tokens = ex.feature_lists.feature_list["answer"]

    for token in context_ids:
        context_tokens.feature.add().int64_list.value.append(token)
    for token in question_ids:
        question_tokens.feature.add().int64_list.value.append(token)
    for token in answer_ids:
        #print(token)
        answer_tokens.feature.add().int64_list.value.append(token)

    return ex

定义写入功能

def write_example_to_tfrecord(context, question, answer, tfrecord_file, writer):
      example= sequence_to_tf_example(context, question, answer)
      writer.write(example.SerializeToString())

def write_data_to_tf_record(filename):
    file_csv= filename+'.csv'
    file_tfrecords= filename+'.tfrecords'
    with open(file_csv) as csvfile:
       readCSV = csv.reader(csvfile, delimiter=',')
       next(readCSV) #skip header
       writer= tf.python_io.TFRecordWriter(file_tfrecords)
       for row in readCSV:
       write_example_to_tfrecord(row[0], row[1], row[2], file_tfrecords, writer)
       writer.close()

定义读取功能

def read_from_tfrecord(ex):

   sequence_features = {
     "context": tf.FixedLenSequenceFeature([], dtype=tf.int64),
     "question": tf.FixedLenSequenceFeature([], dtype=tf.int64),
     "answer": tf.FixedLenSequenceFeature([], dtype=tf.int64)
 }

# Parse the example (returns a dictionary of tensors)
_, sequence_parsed = tf.parse_single_sequence_example(
    serialized=ex,
    sequence_features=sequence_features
)

return {"context": sequence_parsed['context'], "question": sequence_parsed['question'],
        "answer": sequence_parsed['answer']}

创建数据集

def make_dataset(path, batch_size=128):
  '''
  Makes  a Tensorflow dataset that is shuffled, batched and parsed.
  '''
   # Read a tf record file. This makes a dataset of raw TFRecords
   dataset = tf.data.TFRecordDataset([path])
   # Apply/map the parse function to every record. Now the dataset is a bunch of dictionaries of Tensors
   dataset =  dataset.map(read_from_tfrecord)
   #Shuffle the dataset
   dataset = dataset.shuffle(buffer_size=10000)

# specify padding for each tensor seperatly
 dataset = dataset.padded_batch(batch_size, padded_shapes={
    "context": tf.TensorShape([None]), 
    "question": tf.TensorShape([None]), 
    "answer": tf.TensorShape([None]) 
})

return dataset

TFRecords用于嵌入的文本数据

1 个答案: