TensorFlow对象检测API:推理期间内存不足

时间:2019-02-19 09:22:29

标签: tensorflow object-detection object-detection-api

背景:

  • Windows 10
  • Tensorflow 1.12
  • CUDA 9.0
  • CuDNN 7.4.2
  • Python 3.6.8
  • GTX 1080Ti

我已经按照official document进行了设置。我还提到了here,将我的安装验证为sentdex’s Youtube video。在Jupyter Notebook中一切正常。

然后我将代码复制到python文件中,如下所示,在其中进行了较小的更改以适应项目结构。

import sys
import os
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np
from PIL import Image
sys.path.append("[path to program]\\models\\research\\")
sys.path.append("[path to program]\\models\\research\\object_detection\\utils")
from models.research.object_detection.utils import visualization_utils as vis_util
from models.research.object_detection.utils import ops as utils_ops
from models.research.object_detection.utils import label_map_util

def run_inference_for_single_image(image, graph):
  (no any change here)

def load_image_into_numpy_array(image):
  (no any change here)

category_index = label_map_util.create_category_index_from_labelmap('./images/labels/mscoco_label_map.pbtxt', use_display_name=True)

detection_graph = tf.Graph()
with detection_graph.as_default():
  od_graph_def = tf.GraphDef()
  with tf.gfile.GFile("./models/research/object_detection/ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb", 'rb') as fid:
    serialized_graph = fid.read()
    od_graph_def.ParseFromString(serialized_graph)
    tf.import_graph_def(od_graph_def, name='')

images = os.listdir('./images')
for image_name in images:
  image = Image.open('./images/' + image_name)
  image_np = load_image_into_numpy_array(image)
  image_np_expanded = np.expand_dims(image_np, axis=0)
  output_dict = run_inference_for_single_image(image_np, detection_graph)
  vis_util.visualize_boxes_and_labels_on_image_array(
      image_np,
      output_dict['detection_boxes'],
      output_dict['detection_classes'],
      output_dict['detection_scores'],
      category_index,
      instance_masks=output_dict.get('detection_masks'),
      use_normalized_coordinates=True,
      line_thickness=8)
  plt.figure(figsize=(12, 8))
  plt.imshow(image_np)

但是,使用python -m xxx运行脚本时,它会显示

2019-02-19 16:53:20.523470: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-02-19 16:53:20.730182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.7085
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.10GiB
2019-02-19 16:53:20.734460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-19 16:53:21.675377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-19 16:53:21.678080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0
2019-02-19 16:53:21.679419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N
2019-02-19 16:53:21.680882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8788 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-02-19 16:53:37.896876: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 8.58G (9215183360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-19 16:53:42.648674: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 7.72G (8293664256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-19 16:53:42.851675: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 6.95G (7464297472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-19 16:53:42.958418: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 6.26G (6717867520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
 .... (skipping lots of lines)
    2019-02-19 16:53:53.382786: W tensorflow/core/common_runtime/bfc_allocator.cc:271] <allocator contains no memory>
2019-02-19 16:53:53.384612: W tensorflow/core/framework/op_kernel.cc:1250] OP_REQUIRES failed at constant_op.cc:75 : Resource exhausted: OOM when allocating tensor of shape [1,1,512,273] and type float
2019-02-19 16:53:53.388753: E tensorflow/core/common_runtime/executor.cc:623] Executor failed to create kernel. Resource exhausted: OOM when allocating tensor of shape [1,1,512,273] and type float
         [[{{node BoxPredictor_0/ClassPredictor/weights/read/_178__cf__181}} = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [1,1,512,273] values: [[[-0.00129073276 -0.00339978025 0.0160087105...]]]...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
Traceback (most recent call last):
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [1,1,512,273] and type float
         [[{{node BoxPredictor_0/ClassPredictor/weights/read/_178__cf__181}} = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [1,1,512,273] values: [[[-0.00129073276 -0.00339978025 0.0160087105...]]]...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\Project\Python\sudoku\detector\ssd_mobilenet_v1_coco.py", line 88, in <module>
    output_dict = run_inference_for_single_image(image_np, detection_graph)
  File "D:\Project\Python\sudoku\detector\ssd_mobilenet_v1_coco.py", line 49, in run_inference_for_single_image
    feed_dict={image_tensor: np.expand_dims(image, 0)})
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [1,1,512,273] and type float
         [[node BoxPredictor_0/ClassPredictor/weights/read/_178__cf__181 (defined at D:\Project\Python\sudoku\detector\ssd_mobilenet_v1_coco.py:77)  = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [1,1,512,273] values: [[[-0.00129073276 -0.00339978025 0.0160087105...]]]...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'BoxPredictor_0/ClassPredictor/weights/read/_178__cf__181', defined at:
  File "d:\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\Project\Python\sudoku\detector\ssd_mobilenet_v1_coco.py", line 77, in <module>
    tf.import_graph_def(od_graph_def, name='')
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\framework\importer.py", line 442, in import_graph_def
    _ProcessNewOps(graph)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\framework\importer.py", line 234, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3440, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3440, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3299, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "d:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor of shape [1,1,512,273] and type float
         [[node BoxPredictor_0/ClassPredictor/weights/read/_178__cf__181 (defined at D:\Project\Python\sudoku\detector\ssd_mobilenet_v1_coco.py:77)  = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [1,1,512,273] values: [[[-0.00129073276 -0.00339978025 0.0160087105...]]]...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

出什么问题了?

0 个答案:

没有答案