Question

我使用的是0.10，我的训练网内存不足，导致CUDA内存不足。我看到这个问题，optimize_flag设置为fast_run。下面的确切堆栈跟踪和Theano变量是：

[global]
#cnmem = True
floatX = float32

#device = cpu
device = cuda
optimizer = fast_run
#optimizer = None
allow_gc = False
allow_pre_alloc = False

[lib]
#cnmem = 0.9

[nvcc]
fastmath = True

[blas]
ldflags = -llapack -lblas

[cuda]
root=/usr/local/cuda

#[gpuarray]
#preallocate=

1

堆栈跟踪和异常是

调用中的

/usr/local/lib/python2.7/dist-packages/Theano-0.10.0b4-py2.7.egg/theano/compile/function_module.pyc(self，* args，** kwargs） 915 node = self.fn.nodes [self.fn.position_of_error]， 916 thunk = thunk， - ＆GT; 917 storage_map = getattr（self.fn，'storage_map'，None）） 918否则： 919＃老式连接器提出了自己的异常

    /usr/local/lib/python2.7/dist-packages/Theano-0.10.0b4-py2.7.egg/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)
        323         # extra long error message in that case.
        324         pass
    --> 325     reraise(exc_type, exc_value, exc_trace)
        326 
        327 

    /usr/local/lib/python2.7/dist-packages/Theano-0.10.0b4-py2.7.egg/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
        901         try:
        902             outputs =\
    --> 903                 self.fn() if output_subset is None else\
        904                 self.fn(output_subset=output_subset)
        905         except Exception:

    pygpu/gpuarray.pyx in pygpu.gpuarray.pygpu_empty()

    pygpu/gpuarray.pyx in pygpu.gpuarray.array_empty()

    GpuArrayException: cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    Apply node that caused the error: GpuElemwise{add,no_inplace}(GpuDnnConv{algo='small', inplace=True, num_groups=1}.0, InplaceGpuDimShuffle{x,0,x,x}.0)
    Toposort index: 2391
    Inputs types: [GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, (True, False, True, True))]
    Inputs shapes: [(100, 96, 31, 31), (1, 96, 1, 1)]
    Inputs strides: [(369024, 3844, 124, 4), (384, 4, 4, 4)]
    Inputs values: ['not shown', 'not shown']
    Outputs clients: [[GpuElemwise{Composite{(i0 * (i1 + Abs(i1)))}}[]<gpuarray>(GpuArrayConstant{[[[[ 0.5]]]]}, GpuElemwise{add,no_inplace}.0), GpuElemwise{Composite{((i0 * i1) + (i2 * i1 * sgn(i3)))}}[(0, 3)]<gpuarray>(GpuArrayConstant{[[[[ 0.5]]]]}, GpuSplit{4}.2, GpuArrayConstant{[[[[ 0.5]]]]}, GpuElemwise{add,no_inplace}.0)]]

    Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
      File "/usr/local/lib/python2.7/dist-packages/ipykernel/ipkernel.py", line 196, in do_execute
        res = shell.run_cell(code, store_history=store_history, silent=silent)
      File "/usr/local/lib/python2.7/dist-packages/ipykernel/zmqshell.py", line 533, in run_cell
        return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
        interactivity=interactivity, compiler=compiler, result=result)
      File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
        if self.run_code(code, result):
      File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-35-a7b5c0336969>", line 9, in <module>
        predictions=lasagne.layers.get_output(network,{neto['input']:X_O,neti['input']:X_I,netii['input']:X_II})
  File "/usr/local/lib/python2.7/dist-packages/lasagne/layers/helper.py", line 197, in get_output
    all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/lasagne/layers/conv.py", line 340, in get_output_for
activation = conved + self.b.dimshuffle(('x', 0) + ('x',) * self.n)

GpuArrayException：cuMemAlloc：CUDA_ERROR_OUT_OF_MEMORY：内存不足

0 个答案: