tensorflow版本:1.8.0
运行 train.py 后出现 OutOfRangeError 错误。
然而,训练输出将会生成,但似乎根本不起作用。
请帮我解决问题。
命令:
python object_detection/train.py --logtostderr /
--pipeline_config_path=D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\ssd_mobilenet_v1_coco.config /
--train_dir=D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\train
错误:
OutOfRangeError (see above for traceback): FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]
您可以在下面找到更多日志:
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2018-06-05 23:37:04.592661: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
INFO:tensorflow:Restoring parameters from D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\ssd_mobilenet_v1_coco_2017_11_17\model.ckpt
INFO:tensorflow:Restoring parameters from D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\ssd_mobilenet_v1_coco_2017_11_17\model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\train\model.ckpt
INFO:tensorflow:Saving checkpoint to path D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\train\model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Caught OutOfRangeError. Stopping Training. FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]
Caused by op 'prefetch_queue_Dequeue', defined at:
File "object_detection/train.py", line 184, in <module>
tf.app.run()
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 275, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, **kwargs)
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 177, in _create_losses
train_config.use_multiclass_scores)
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 122, in get_inputs
read_data_list = input_queue.dequeue()
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\core\batcher.py", line 111, in dequeue
batched_tensors = self._queue.dequeue()
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 440, in dequeue
self._queue_ref, self._dtypes, name=name)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 4096, in queue_dequeue_v2
timeout_ms=timeout_ms, name=name)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
op_def=op_def)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
OutOfRangeError (see above for traceback): FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]
INFO:tensorflow:Caught OutOfRangeError. Stopping Training. FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]
Caused by op 'prefetch_queue_Dequeue', defined at:
File "object_detection/train.py", line 184, in <module>
tf.app.run()
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 275, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, **kwargs)
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 177, in _create_losses
train_config.use_multiclass_scores)
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 122, in get_inputs
read_data_list = input_queue.dequeue()
File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\core\batcher.py", line 111, in dequeue
batched_tensors = self._queue.dequeue()
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 440, in dequeue
self._queue_ref, self._dtypes, name=name)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 4096, in queue_dequeue_v2
timeout_ms=timeout_ms, name=name)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
op_def=op_def)
File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
OutOfRangeError (see above for traceback): FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]
INFO:tensorflow:Finished training! Saving model to disk.
INFO:tensorflow:Finished training! Saving model to disk.
生成输出只需要一分钟,我认为这是完成培训过程的正常时间。