Question

我正在学习如何使用tf.data.Dataset api。我正在使用Google为他们的coursera tensorflow类提供的示例代码。具体来说，我正在使用c_dataset.ipynb笔记本here.

此笔记本有一个model.train例程，如下所示：

model.train(input_fn = get_train(), steps = 1000)

get_train（）例程最终使用带有tf.data.Dataset api的代码段调用该代码：

filenames_dataset = tf.data.Dataset.list_files(filename)
# read lines from text files 
# this results in a dataset of textlines from all files
textlines_dataset = filenames_dataset.flat_map(tf.data.TextLineDataset)
# Parse text lines as comma-separated values (CSV)
# this does the decoder function for each textline
dataset = textlines_dataset.map(decode_csv)

这些评论很好地解释了会发生什么。稍后，该例程将像这样返回：

    # return the features and label as a tensorflow node, these
    # will trigger file load operations progressively only when
    # needed.
    return dataset.make_one_shot_iterator().get_next()

总有一次评估结果吗？我尝试过类似的操作，但失败了。

# Try to read what its using from the cvs file.
one_batch_the_csv_file = get_train()

with tf.Session() as sess:
  result = sess.run(one_batch_the_csv_file)

print(one_batch_the_csv_file)

根据下面鲁本的建议，我添加了这个

我继续学习本课中的下一组实验，他们介绍了张量板，我得到了一些图形，但仍然没有输入或输出。话虽如此，这里是一套更完整的资源。

# curious he did not do this
# I am guessing because the output is so verbose
tf.logging.set_verbosity(tf.logging.INFO)  # putting back in since, tf.train.LoggingTensorHook mentions it

def train_and_evaluate(output_dir, num_train_steps):

  # Added this while trying to get input vals from csv.
  # This gives an error about scafolding
  # summary_hook = tf.train.SummarySaverHook()
  # SAVE_EVERY_N_STEPS,
  #  summary_op=tf.summary.merge_all())



  # To convert a model to distributed train and evaluate do four things
  estimator = tf.estimator.DNNClassifier(                         # 1. Estimator
                          model_dir = output_dir,
                          feature_columns = feature_cols, 
                          hidden_units=[160, 80, 40, 20],
                          n_classes=2,
                          config=tf.estimator.RunConfig().replace(save_summary_steps=2)  # 2. run config
                                                                                          # ODD. he mentions we need a run config in the videos, but it was missing in the lab
                                                                                          # notebook.  Later I found the bug report which gave me this bit of code.
                                                                                          # I got a working TensorBoard when I changed this from save_summary_steps=10 to 2.
                          )# 


  # .. also need the trainspec to tell the estimator how to get training data
  train_spec = tf.estimator.TrainSpec(
                                    input_fn = read_dataset('./taxi-train.csv', mode = tf.estimator.ModeKeys.TRAIN), # make sure you use the dataset api
                                    max_steps = num_train_steps)
#                                    training_hook=[summary_hook])       # Added this while trying to get input vals from csv.

  # ... also need this
  # serving and training-time inputs are often very different
  exporter = tf.estimator.LatestExporter('exporter', serving_input_receiver_fn = serving_input_fn)




  # .. also need an EvalSpec which controls the evaluation and
  # the checkpointing of the model since they happen at the same time
  eval_spec = tf.estimator.EvalSpec(
                                    input_fn = read_dataset('./taxi-valid.csv', mode = tf.estimator.ModeKeys.EVAL), # make sure you use the dataset api
                                    steps=None, # evals on 100 batches
                                    start_delay_secs = 1, # start evaluating after N secoonds. orig was 1.  3 seemed to fail?
                                    throttle_secs = 10, # eval no more than every 10 secs. Can not be more frequent than the checkpoint config specified in the run config.
                                    exporters = exporter)  # how to export the model for production.

  tf.estimator.train_and_evaluate(
                                estimator,
                                train_spec,  # 3. Train Spec
                                eval_spec)   # 4. Eval Spec


OUTDIR = './model_trained'
shutil.rmtree(OUTDIR, ignore_errors = True) # start fresh each time
TensorBoard().start(OUTDIR)
# need to let this complete before running next cell 


# call the above routine
train_and_evaluate(OUTDIR, num_train_steps = 6000)  # originally 2000. 1000 after reset shows only projectors

Answer 1

我不确定您要提取哪种信息。如果您对步骤N感兴趣，请作为一般答案：

如果要获得准确的结果，只需运行model.train(input_fn = get_train(), steps = N)。
在确定的步骤中检查火车模块功能here的具体内容。

如果您搜索步骤，则会发现不同的类：

CheckpointSaverHook：每N步或每秒钟保存检查点。
LoggingTensorHook：每N个局部步长，每N秒或最后打印给定张量。
ProfilerHook：每N步或每秒钟捕获一次CPU / GPU分析信息。
SummarySaverHook：每N个步骤保存摘要。
等等（还有更多，只需检查对您有用的内容）。

如何检查基于tensorflow.data.Dataset的model.train input_fn的结果

1 个答案: