启动TensorFlow会话时出现“ CUDA_ERROR_INVALID_DEVICE:无效的设备顺序”

时间:2019-07-18 07:53:25

标签: tensorflow nvidia cudnn

启动TensorFlow会话时,未检测到GPU(CUDA_ERROR_INVALID_DEVICE: invalid device ordinal):

$ CUDA_VISIBLE_DEVICES='0' python3 -c 'import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))'
2019-07-18 09:36:55.661519: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-18 09:36:55.684438: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3312000000 Hz
2019-07-18 09:36:55.684721: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x41adbb0 executing computations on platform Host. Devices:
2019-07-18 09:36:55.684750: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-18 09:36:55.686513: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-18 09:36:55.696958: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
2019-07-18 09:36:55.697001: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: tobias-Z170-HD3P
2019-07-18 09:36:55.697006: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: tobias-Z170-HD3P
2019-07-18 09:36:55.697084: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 410.73.0
2019-07-18 09:36:55.697108: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 410.73.0
2019-07-18 09:36:55.697113: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 410.73.0
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
2019-07-18 09:36:55.697380: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device

使用CUDA_VISIBLE_DEVICES='1'也无济于事。

Cuda(cuda_10.0.130_410.48_linux.run)已安装。

$ cat /usr/local/cuda/version.txt
CUDA Version 10.0.130

CuDNN(cudnn-10.0-linux-x64-v7.4.2.24.tgz):

$ cat /usr/include/x86_64-linux-gnu/cudnn_v*.h | grep CUDNN_MAJOR -A 2 | head -n 3
#define CUDNN_MAJOR      6
#define CUDNN_MINOR      0
#define CUDNN_PATCHLEVEL 21

TensorFlow(pip3 install tensorflow-gpu):

$ python3 -c 'import tensorflow as tf; print(tf.__version__)'
1.14.0

Nvidia驱动程序(NVIDIA-Linux-x86_64-410.73.run)也可以:

$ nvidia-smi 
Thu Jul 18 09:35:03 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.73       Driver Version: 410.73       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   45C    P8    17W / 230W |    569MiB /  8111MiB |     19%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2270      G   /usr/lib/xorg/Xorg                           301MiB |
|    0      3021      G   /opt/zoom/zoom                                14MiB |
|    0      3503      G   ...-token=CB875E52FAB2279C6A34C6519188AD9C    71MiB |
|    0      3534      G   ...uest-channel-token=16121978823314344450    56MiB |
|    0      3618      G   ...uest-channel-token=12369473663213430887    52MiB |
|    0      4249      G   ...uest-channel-token=13759302641460814281    62MiB |
|    0      4499      G   ...uest-channel-token=10576172133955227583     7MiB |
+-----------------------------------------------------------------------------+

我正在使用Linux Mint 18.2。

有什么想法吗?

1 个答案:

答案 0 :(得分:0)

解决了。我卸载了从NVIDIA-Linux-x86_64-410.73.run安装的突触软件包管理器中显示的所有Nvidia驱动程序版本,现在可以了。

记录:使用命令行卸载可能如下所示:

sudo nvidia-uninstall
sudo apt-get remove --purge nvidia-*
$ python3 -c 'import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))'

2019-07-18 10:57:07.020764: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-18 10:57:07.059271: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3312000000 Hz
2019-07-18 10:57:07.060038: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x53aec90 executing computations on platform Host. Devices:
2019-07-18 10:57:07.060060: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-18 10:57:07.069543: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-18 10:57:07.216124: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.216596: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54792a0 executing computations on platform CUDA. Devices:
2019-07-18 10:57:07.216612: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
2019-07-18 10:57:07.216803: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.217224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
2019-07-18 10:57:07.218763: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-18 10:57:07.243155: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-18 10:57:07.257961: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-18 10:57:07.263297: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-18 10:57:07.298517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-18 10:57:07.321558: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-18 10:57:07.394510: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-18 10:57:07.394806: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.396131: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.397206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-18 10:57:07.397798: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-18 10:57:07.400997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-18 10:57:07.401041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2019-07-18 10:57:07.401059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2019-07-18 10:57:07.401572: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.402874: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-18 10:57:07.404129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7060 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1
2019-07-18 10:57:07.405492: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1