Does training tensorflow model automatically save parameters?

Question

I ran the demo tensorflow MNIST model(in models/image/mnist) by

python -m tensorflow.models.image.mnist.convolutional

Does it mean that after the model completes training, the parameters/weights are automatically stored on secondary storage? Or do we have to edit the code to include "saver" functions for parameters to be stored?

Answer 1

No they are not automatically saved. Everything is in memory. You have to explicitly add a saver function to store your model to a secondary storage.

First you create a saver operation

saver = tf.train.Saver(tf.all_variables())

Then you want to save your model as it progresses in the train process, usually after N steps. This intermediate steps are commonly named "checkpoints".

  # Save the model checkpoint periodically.
  if step % 1000 == 0:
    checkpoint_path = os.path.join('.train_dir', 'model.ckpt')
    saver.save(sess, checkpoint_path)

Then you can restore the model from the checkpoint:

 saver.restore(sess, model_checkpoint_path)

Take a look at tensorflow.models.image.cifar10 for a concrete example