在TPU中使用SessionRunHook

时间:2018-12-19 11:05:16

标签: tensorflow tpu

我已经开发了一个SessionRunHook,并将其附加到TPUEstimator。 SessionRunHook在CPU上可以正常工作,但是如果我使用TPU,则会出现错误:

INFO:tensorflow:Error recorded from outfeed: Attempted to use a closed Session.
INFO:tensorflow:Error recorded from evaluation_loop: Operation 'mean_1' has been marked as not fetchable.
INFO:tensorflow:evaluation_loop marked as finished
WARNING:tensorflow:Reraising captured error
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 549, in <module>
    tf.app.run()
INFO:tensorflow:Error recorded from infeed: Step was cancelled by an explicit call to `Session::Close()`.
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 505, in main
    result = estimator.evaluate(input_fn=input_fn, steps=FLAGS.max_eval_steps)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2424, in evaluate
    rendezvous.raise_errors()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 128, in raise_errors
    six.reraise(typ, value, traceback)
  File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 101, in catch_errors
    yield
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 451, in _run_outfeed
    session.run(self._dequeue_ops)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1075, in _run
    raise RuntimeError('Attempted to use a closed Session.')
RuntimeError: Attempted to use a closed Session.

还有其他人遇到此问题并找到解决方法吗?

0 个答案:

没有答案