Tensorflow“尝试分配4B而内存不足”

时间:2017-03-28 23:36:13

标签: tensorflow gpu

我在没有GPU支持的笔记本电脑上运行TensorFlow的小型网络。我现在正试图在带有GPU的Linux机器上运行它,我相信我已正确安装了所有内容。 (当然TensorFlow的“Hello,world!”示例运行。)

但是,当我尝试运行我的程序时,我收到一个以这条神秘线开头的错误:

W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 4B.  See logs for memory state.

真的? 4字节?我不认为我的GPU 小。

我有两个理论,但没有真正的线索如何继续:

  1. 某些安装细节或命令行选项未正确设置,因此我的程序为GPU分配了0个内存。
  2. 我在我的程序中犯了一些错误,在没有GPU的情况下运行时没有出现问题,但在我拥有GPU时崩溃。
  3. 这是输出的较长版本:

    I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
    I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
    I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
    I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
    I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
    inputs (1619, 691200)
    <OUTPUT SPECIFIC TO MY PROGRAM APPEARS HERE>
    W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
    W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
    W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
    W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
    W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
    W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
    I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
    name: GeForce GTX 650
    major: 3 minor: 0 memoryClockRate (GHz) 1.0585
    pciBusID 0000:01:00.0
    Total memory: 973.69MiB
    Free memory: 796.44MiB
    I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
    I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
    I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 650, pci bus id: 0000:01:00.0)
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
    I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 256B was 256B, Chunk State: 
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x500b80000 of size 1280
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x500b80500 of size 256
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x500b80600 of size 512
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x500b80800 of size 2764800
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x500e23800 of size 256
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x500e23900 of size 256
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x500e23a00 of size 276480000
    I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x5115cfa00 of size 346162688
    I tensorflow/core/common_runtime/bfc_allocator.cc:693]      Summary of in-use Chunks by size: 
    I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 256 totalling 768B
    I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 512 totalling 512B
    I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
    I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 2764800 totalling 2.64MiB
    I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 276480000 totalling 263.67MiB
    I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 346162688 totalling 330.13MiB
    I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 596.44MiB
    I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: 
    Limit:                   625410048
    InUse:                   625410048
    MaxInUse:                625410048
    NumAllocs:                       8
    MaxAllocSize:            346162688
    
    W tensorflow/core/common_runtime/bfc_allocator.cc:274] *****************************************************************************************xxxxxxxxxxx
    W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 4B.  See logs for memory state.
    W tensorflow/core/framework/op_kernel.cc:983] Internal: Dst tensor is not initialized.
    E tensorflow/core/common_runtime/executor.cc:594] Executor failed to create kernel. Internal: Dst tensor is not initialized.
         [[Node: random_uniform/sub/_1__cf__1 = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [] values: 2>, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
    Traceback (most recent call last):
      File "/home/drake/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1022, in _do_call
        return fn(*args)
      File "/home/drake/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1004, in _run_fn
        status, run_metadata)
      File "/home/drake/anaconda3/lib/python3.6/contextlib.py", line 89, in __exit__
        next(self.gen)
      File "/home/drake/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
        pywrap_tensorflow.TF_GetCode(status))
    tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
         [[Node: random_uniform/sub/_1__cf__1 = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [] values: 2>, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "tinyAE.py", line 55, in <module>
        sess.run(tf.global_variables_initializer())
      File "/home/drake/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 767, in run
        run_metadata_ptr)
      File "/home/drake/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 965, in _run
        feed_dict_string, options, run_metadata)
      File "/home/drake/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
        target_list, options, run_metadata)
      File "/home/drake/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
        raise type(e)(node_def, op, message)
    tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
         [[Node: random_uniform/sub/_1__cf__1 = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [] values: 2>, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
    

0 个答案:

没有答案