Tensorflow Training过程结束时出现错误:OutOfRangeError(参见上面的回溯):FIFOQueue' _3_prefetch_queue'

时间:2018-06-05 19:47:39

标签: python tensorflow

tensorflow版本:1.8.0

运行 train.py 后出现 OutOfRangeError 错误。

然而,训练输出将会生成,但似乎根本不起作用。

请帮我解决问题。

命令:

    python object_detection/train.py --logtostderr /
            --pipeline_config_path=D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\ssd_mobilenet_v1_coco.config /
            --train_dir=D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\train

错误:

OutOfRangeError (see above for traceback): FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
         [[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]
您可以在下面找到更多日志:

Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2018-06-05 23:37:04.592661: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
INFO:tensorflow:Restoring parameters from D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\ssd_mobilenet_v1_coco_2017_11_17\model.ckpt
INFO:tensorflow:Restoring parameters from D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\ssd_mobilenet_v1_coco_2017_11_17\model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\train\model.ckpt
INFO:tensorflow:Saving checkpoint to path D:\Documents\me\GitiBon\Projects\GLOVE\TensorFlow\Train\models\GitibonGlove\train\model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Caught OutOfRangeError. Stopping Training. FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
         [[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]

Caused by op 'prefetch_queue_Dequeue', defined at:
  File "object_detection/train.py", line 184, in <module>
    tf.app.run()
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
    _sys.exit(main(argv))
  File "object_detection/train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 275, in train
    clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\slim\deployment\model_deploy.py", line 193, in create_clones
    outputs = model_fn(*args, **kwargs)
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 177, in _create_losses
    train_config.use_multiclass_scores)
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 122, in get_inputs
    read_data_list = input_queue.dequeue()
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\core\batcher.py", line 111, in dequeue
    batched_tensors = self._queue.dequeue()
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 440, in dequeue
    self._queue_ref, self._dtypes, name=name)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 4096, in queue_dequeue_v2
    timeout_ms=timeout_ms, name=name)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
    op_def=op_def)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
         [[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]

INFO:tensorflow:Caught OutOfRangeError. Stopping Training. FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
         [[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]

Caused by op 'prefetch_queue_Dequeue', defined at:
  File "object_detection/train.py", line 184, in <module>
    tf.app.run()
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
    _sys.exit(main(argv))
  File "object_detection/train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 275, in train
    clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\slim\deployment\model_deploy.py", line 193, in create_clones
    outputs = model_fn(*args, **kwargs)
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 177, in _create_losses
    train_config.use_multiclass_scores)
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\trainer.py", line 122, in get_inputs
    read_data_list = input_queue.dequeue()
  File "D:\Documents\me\GitiBon\Projects\GLOVE\TENSORFLOW\models-master\research\object_detection\core\batcher.py", line 111, in dequeue
    batched_tensors = self._queue.dequeue()
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 440, in dequeue
    self._queue_ref, self._dtypes, name=name)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 4096, in queue_dequeue_v2
    timeout_ms=timeout_ms, name=name)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
    op_def=op_def)
  File "C:\Users\mahdi\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_3_prefetch_queue' is closed and has insufficient elements (requested 1, current size 0)
         [[Node: prefetch_queue_Dequeue = QueueDequeueV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prefetch_queue)]]

INFO:tensorflow:Finished training! Saving model to disk.
INFO:tensorflow:Finished training! Saving model to disk.

生成输出只需要一分钟,我认为这是完成培训过程的正常时间。

0 个答案:

没有答案