Question

我在Cupy方面取得了进步，并且能够将以前使用numpy和C ++的迭代图像重建加速大约3倍。现在我遇到一个间歇性问题，指出我似乎错过了一些重要的事情。

我正在使用NVIDIA GeForce GTX 1080 Ti在Mac Pro 2013后期OSX 10.13.6上运行此程序。

我受到内存的限制，并继续使用cupy.asarray（）将数据加载到GPU，然后通过将变量设置为None将其删除。这有效，但间歇性地我得到： cupy.cuda.memory.OutOfMemoryError

我在迭代中运行以下循环：

# iterations
for i in range(nr_iterations):
    [...]

    # loop within iterations
    for idx in range(dim2):
        print("GPU memory info - iteration: " + str(i) + " - used: " + str(mempool.used_bytes()) + ", total: " + str(mempool.total_bytes()) + " pinned :" + str(pinned_mempool.n_free_blocks()) + " in loop with idx = " + str(idx) )
        cupy_array = cp.asarray( cpp_function(numpy_array[:,flow_idx:flow_idx+1,...]) )

        # do all the work
        [...]

        # last line in for loop
        cupy_array = None

打印输出显示稳定的内存使用，直到突然它在 cupy_array = cp.asarray（）的行上引发错误。这里是记录语句的摘录：

GPU memory info - iteration: 0 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 0
GPU memory info - iteration: 0 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 1
GPU memory info - iteration: 1 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 0
GPU memory info - iteration: 1 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 1
GPU memory info - iteration: 2 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 0
GPU memory info - iteration: 2 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 1
GPU memory info - iteration: 3 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 0
GPU memory info - iteration: 3 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 1
GPU memory info - iteration: 4 - used: 3477175296, total: 8529552896 pinned :2 in loop with idx = 0
File "cupy/core/core.pyx", line 1712, in cupy.core.core.array
File "cupy/core/core.pyx", line 1751, in cupy.core.core.array
File "cupy/core/core.pyx", line 134, in cupy.core.core.ndarray.__init__
File "cupy/cuda/memory.pyx", line 518, in cupy.cuda.memory.alloc
File "cupy/cuda/memory.pyx", line 1085, in cupy.cuda.memory.MemoryPool.malloc
File "cupy/cuda/memory.pyx", line 1106, in cupy.cuda.memory.MemoryPool.malloc
File "cupy/cuda/memory.pyx", line 934, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc
File "cupy/cuda/memory.pyx", line 949, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
File "cupy/cuda/memory.pyx", line 697, in cupy.cuda.memory._try_malloc
cupy.cuda.memory.OutOfMemoryError: out of memory to allocate 2158119936 bytes (total 10687672832 bytes)

令我特别惊讶的是，在发生错误的情况下，系统似乎需要10687672832字节，日志记录表明已使用3477175296字节，而分配了2158119936字节。虽然我意识到需要大约两倍的可用内存来加载数组，但在这种情况下，实际上需要的内存是要加载的数组大小的3倍以上。

有什么明显的我想念的东西吗？谢谢您的关注。

Cupy中的间歇性OutOfMemoryError

0 个答案: