Question

我正在尝试使用Tensorflow估计器API在AI引擎上针对DNN回归器进行超参数调整。但是提交作业后，它表明该作业失败了，并且在作业详细信息中出现了此错误。

有人可以帮忙吗？

Hyperparameter Tuning Trial #1 Failed before any other successful trials were completed. The failed trial had parameters: learning_rate=0.0019937718716419557, num-layers=2, first-layer-size=148, scale-factor=0.7910729020312588, .  The trial's error message was: The replica master 0 exited with a non-zero status of 1. 
  Traceback (most recent call last):
    [...]
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 507, in _build_internal
      restore_sequentially, reshape)
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 385, in _AddShardedRestoreOps
      name="restore_shard"))
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
      restore_sequentially)
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
      return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
      name=name)
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
      op_def=op_def)
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
      op_def=op_def)
    File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
      self._traceback = tf_stack.extract_stack()

  InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

  tensor_name = dnn/hiddenlayer_0/bias; shape in shape_and_slice spec [148] does not match the shape stored in checkpoint: [117]
     [[node save/RestoreV2_1 (defined at /usr/local/lib/python3.5/dist-packages/tensorflow_estimator/python/estimator/estimator.py:1403) ]]

Answer 1

看起来您对所有试验都使用相同的输出目录，因此trial＃1试图读取trial＃2检查点（可能是因为它是目录中的最新检查点），但由于体系结构不同而失败< / p>

确保每次超级参数训练运行都使用不同的输出目录。您可以通过两种方式进行此操作：

使用--job-dir作为输出目录。
将hyperparam试用编号添加到您现在使用的输出目录中：

output_dir = os.path.join（output_dir，json.loads（os.environ.get（'TF_CONFIG'，'{}'））.get（'task'，{}）。get（'trial'， ''））

我收到错误消息：“从检查点还原失败。”在AI平台上训练Tensorflow估算器API时（ml-engine）

1 个答案: