FailedPreconditionError：从容器读取资源变量***时出错

Question

我在运行带有horovod回调的model.fit时看到以下错误。如果我跳过回调，model.fit运行正常。注意：我正在使用horovod.tensorflow.keras软件包，而我的模型是基于tensorflow.keras的（我不是使用keras软件包，而是从tensorflow中获得了成功）

FailedPreconditionError: Error while reading resource variable conv1d/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/conv1d/kernel/N10tensorflow3VarE does not exist.
     [[{{node conv1d/conv1d/ExpandDims_1/ReadVariableOp}}]]

回调如下

callbacks = [
    # Horovod: broadcast initial variable states from rank 0 to all other processes.
    # This is necessary to ensure consistent initialization of all workers when
    # training is started with random weights or restored from a checkpoint.
    hvd.callbacks.BroadcastGlobalVariablesCallback(0),
    TensorBoard(log_dir='boardlogs/{}'.format(datetime.datetime.now())) #report logs to tensorboard
]
# Horovod: save checkpoints only on worker 0 to prevent other workers from corrupting them.
if hvd.rank() == 0:
    horovod_callbacks.append(tf.keras.callbacks.ModelCheckpoint('.horovod-cps/checkpoint-{epoch}.h5')


history = model.fit(X, y, epochs=500, batch_size=64, callbacks=callbacks, verbose=1 if hvd.rank() == 0 else 0)

环境： 框架：tensorflow.keras Tensorflow版本1.13.1 Keras版本2.2.4-tf Horovod版本：horovod == 0.17.0.post1 Python版本：3.6