CUDA_ERROR_OUT_OF_MEMORY:内存不足:对于tensorflow 2.1

时间:2020-04-23 08:08:30

标签: python tensorflow deep-learning tensorflow2.0

我是tensorflow-gpu的新手,在CPU上运行似乎还不错,但无法以某种方式使GPU版本正常工作。请让我知道下一步该怎么做。非常感谢!

我在TensorFlow 2.1中使用Python 3.7.7,并使用

安装了它
conda install tensorflow-gpu

系统规格:

Intel(R) core(TM) I5-7440HQ CPU @ 2.80 GHZ
RAM: 8GB

GPU规格:

Model: GeForce 930MX
GPU memory: 5.9 GB
Dedicated GPU memory: 2GB
Shared GPU memory: 3.9 GB

NVIDIA-SMI

 NVIDIA-SMI 445.87       Driver Version: 445.87       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 930MX      WDDM  | 00000000:02:00.0 Off |                  N/A |
| N/A   49C    P8    N/A /  N/A |     37MiB /  2048MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU                  PID   Type   Process name                  GPU Memory |
|                                                                  Usage      |

以批处理大小32运行简单的MNIST数据集训练。

Jupyter笔记本命令提示符:

2020-04-23 12:43:12.448744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
    2020-04-23 12:43:18.625257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
    2020-04-23 12:43:18.863674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
    pciBusID: 0000:02:00.0 name: GeForce 930MX computeCapability: 5.0
    coreClock: 1.0195GHz coreCount: 3 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 13.41GiB/s
    2020-04-23 12:43:18.869948: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
    2020-04-23 12:43:18.943177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
    2020-04-23 12:43:19.004099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
    2020-04-23 12:43:19.030424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
    2020-04-23 12:43:19.092306: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
    2020-04-23 12:43:19.139074: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
    2020-04-23 12:43:19.264762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
    2020-04-23 12:43:19.436399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
    2020-04-23 12:43:19.443444: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
    2020-04-23 12:43:19.455503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
    pciBusID: 0000:02:00.0 name: GeForce 930MX computeCapability: 5.0
    coreClock: 1.0195GHz coreCount: 3 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 13.41GiB/s
    2020-04-23 12:43:19.463043: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
    2020-04-23 12:43:19.467340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
    2020-04-23 12:43:19.470920: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
    2020-04-23 12:43:19.477116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
    2020-04-23 12:43:19.486208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
    2020-04-23 12:43:19.494696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
    2020-04-23 12:43:19.505751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
    2020-04-23 12:43:19.515014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
    2020-04-23 12:43:24.165525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
    2020-04-23 12:43:24.169026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0
    2020-04-23 12:43:24.171068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N
    2020-04-23 12:43:24.175336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1377 MB memory) -> physical GPU (device: 0, name: GeForce 930MX, pci bus id: 0000:02:00.0, compute capability: 5.0)
    2020-04-23 12:43:24.219369: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.34G (1444337920 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.237697: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.21G (1299904256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.260040: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.09G (1169913856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.284695: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1004.14M (1052922624 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.306355: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 903.73M (947630336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.327752: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 813.36M (852867328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.357554: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 732.02M (767580672 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.384318: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 658.82M (690822656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.406377: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 592.94M (621740544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    2020-04-23 12:43:24.426737: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 533.64M (559566592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
    [I 12:43:26.566 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports

以下是我要训练的模型。在CPU上正常工作。

model = Sequential()

model.add(Conv2D(256, (3, 3), input_shape=X.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(256, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors

model.add(Dense(64))
model.add(Activation('relu'))

model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(X, y, batch_size=32, epochs=3, validation_split=0.3)

1 个答案:

答案 0 :(得分:0)

Cuda11.0与Tensorflow2.1不兼容。请检查兼容性here

版本Python版本编译器构建工具cuDNN CUDA。
tensorflow-2.1.0 2.7,3.5-3.7 GCC 7.3.1 Bazel 0.27.1 7.6 10.1

Tensorflow 2.1与Cuda10.1兼容。所以你有两个选择

选项1

创建一个conda环境并安装tensorflow-gpu == 2.1

conda create -n tf_gpu
source activete tf_gpu
Within the virtual environment
conda install tensorflow-gpu=2.1

有时以下工作

conda create --name <some_name> tensorflow-gpu=2.1.0 cudatoolkit=10.1 python=3.6

选项2

使用上述命令(用于基于conda的安装)卸载Tensorflow和Cuda11.0,关闭并重新启动计算机,然后重新安装tensorflow-gpu,或按照说明here使用pip进行安装。