Question

在GPU上运行keras时收到此错误。

  File "modeltra.py", line 165, in <module>
    x_train = a_model.predict(y_training)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/engine/training.py", line 1169, in predict
    steps=steps)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2697, in __call__
    if hasattr(get_session(), '_make_callable_from_options'):
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 186, in get_session
    _SESSION = tf.Session(config=config)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1570, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 693, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

我的GPU信息如下

(gp_ker_pip) [hgam@xx]$ nvidia-smi
Tue Aug 27 17:29:34 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  On   | 00000000:03:00.0 Off |                    0 |
| N/A   34C    P0    33W / 250W |    855MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  On   | 00000000:82:00.0 Off |                    0 |
| N/A   32C    P0    31W / 250W |    855MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

(gp_ker_pip) [hgam@xx]$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

首先使用 pip install tensorflow-gpu

并安装了keras conda install keras-gpu

我应该遵循什么才能完成这项工作？另外如何检查CUDA运行时版本和驱动程序版本？

PS：我还使用

降级了cuda版本

>    export PATH=$PATH:/usr/local/cuda-9.0/bin
>    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.0/lib64

仍然出现错误。

Answer 1

按照罗伯特的建议安装CUDA 10.1后，此问题已解决。

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:03:00.0 Off |                    0 |
| N/A   39C    P0    64W /  70W |  14503MiB / 15079MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:82:00.0 Off |                    0 |
| N/A   35C    P0    25W /  70W |    111MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

CUDA驱动程序版本不足以用于CUDA运行时版本CUDA 10.0版本

1 个答案: