创建会话后初始化期间出现以下错误
"ValueError: Operation 'Loss_Optimization/cond_1/report_uninitialized_variables/boolean_mask/GatherV2' has been marked as not fetchable"
当我进行多次gpu塔训练时,看不到以上错误。我进行分布式培训时会发生错误。可能是什么原因?另外,我如何设法找出构建此操作的代码行。
下面是详细的错误消息
1411.master | [2019-05-30T21:45:25Z] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 807, in create_session
1411.master | [2019-05-30T21:45:25Z] hook.after_create_session(self.tf_sess, self.coord)
1411.master | [2019-05-30T21:45:25Z] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/sync_replicas_optimizer.py", line 488, in after_create_session
1411.master | [2019-05-30T21:45:25Z] "Model is not ready for SyncReplicasOptimizer local init.")
1411.master | [2019-05-30T21:45:25Z] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/session_manager.py", line 525, in _ready
1411.master | [2019-05-30T21:45:25Z] ready_value = sess.run(op)
1411.master | [2019-05-30T21:45:25Z] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
1411.master | [2019-05-30T21:45:25Z] run_metadata_ptr)
1411.master | [2019-05-30T21:45:25Z] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1137, in _run
1411.master | [2019-05-30T21:45:25Z] self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)
1411.master | [2019-05-30T21:45:25Z] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 484, in __init__
1411.master | [2019-05-30T21:45:25Z] self._assert_fetchable(graph, fetch.op)
1411.master | [2019-05-30T21:45:25Z] File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 497, in _assert_fetchable
1411.master | [2019-05-30T21:45:25Z] 'Operation %r has been marked as not fetchable.' % op.name)
1411.master | [2019-05-30T21:45:25Z] ValueError: Operation 'Loss_Optimization/cond_1/report_uninitialized_variables/boolean_mask/GatherV2' has been marked as not fetchable.