训练Yolo v3模型时出现ResourceExhaustedError

时间:2020-06-19 19:07:05

标签: tensorflow deep-learning yolo

我正在使用183个图像和批注进行袋鼠检测(https://github.com/experiencor/keras-yolo3)的数据集从头开始训练keras-yolo3(https://github.com/experiencor/kangaroo),并支持GPU(Quadro T2000)和32 GB RAM。该训练模型包含约61 M个参数。

输出错误消息是:

    ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[4,64,208,208] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node training_4/Adam/gradients/zeros_273}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[loss_4/Identity_2/_3879]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[4,64,208,208] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node training_4/Adam/gradients/zeros_273}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

我不知道形状[4,64,208,208]的张量如何导致内存不足(OOM)。如何解决?

0 个答案:

没有答案