How to restore a smaller variable to larger variable in TensorFlow?

Question

I made a previously trained model that classifies sentences.

I want to make some variables bigger, but want to restore the old variables which is smaller and the rest part of the variables to be initialized new.

Here is an image of what I want:

When I tried this, this error occurred.

InvalidArgumentError (see above for traceback): Assign requires shapes
of both tensors to match. lhs shape= [13173,32] rhs shape= [13113,32]
[[Node: save_1/Assign = Assign[T=DT_FLOAT,
 _class=["loc:@embedding/embedding_W"], use_locking=true, validate_shape=true,
_device="/job:localhost/replica:0/task:0/gpu:0"](embedding/embedding_W,
save_1/RestoreV2/_5)]]

Answer 1

You cannot restore smaller variables to larger variables directly for several reasons, for example TensorFlow does not know what part of the larger variable the smaller variable should be embedded in.

The way to do this properly is instead to load the old variable, create the new variable and then assign the old variable to a subset of the new variables.

In code, that would be:

# the old variable, as it was before
old_variable = tf.Variable(..., name='old_name')  

# variable with new shape, the one you want to use now
new_variable = tf.Variable(...) 

# Initialize the variables and restore the checkpoint
sess.run(tf.global_variables_initializer())
saver.restore(sess, 'old_checkpoint')

# assign op for the places you want to fill in the old variable
idx = tuple(slice(0, osi) for osi in old_variable.shape)
asssign_op = new_variable[idx].assign(old_variable) 
sess.run(asssign_op)