我已经开发了一个SessionRunHook,并将其附加到TPUEstimator。 SessionRunHook在CPU上可以正常工作,但是如果我使用TPU,则会出现错误:
INFO:tensorflow:Error recorded from outfeed: Attempted to use a closed Session.
INFO:tensorflow:Error recorded from evaluation_loop: Operation 'mean_1' has been marked as not fetchable.
INFO:tensorflow:evaluation_loop marked as finished
WARNING:tensorflow:Reraising captured error
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 549, in <module>
tf.app.run()
INFO:tensorflow:Error recorded from infeed: Step was cancelled by an explicit call to `Session::Close()`.
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 505, in main
result = estimator.evaluate(input_fn=input_fn, steps=FLAGS.max_eval_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2424, in evaluate
rendezvous.raise_errors()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 128, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 101, in catch_errors
yield
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 451, in _run_outfeed
session.run(self._dequeue_ops)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1075, in _run
raise RuntimeError('Attempted to use a closed Session.')
RuntimeError: Attempted to use a closed Session.
还有其他人遇到此问题并找到解决方法吗?