tensorflow：找不到dnn实现

时间：2019-05-27 23:24:52

标签： python tensorflow gpu nvidia cudnn

我试图使用gpu在tensorflow上运行我的代码Keras CuDNNGRU，但即使我已经安装了CUDA和CuDNN，它始终会出现错误“无法找到dnn实现”。

我已经多次重新安装CUDA和CuDNN，并将CuDNN版本从7.2.1升级到7.5.0，但是它没有解决任何问题。我也尝试在Jupyter Notebook和python编译器（在终端上）上运行我的代码，并且两者的结果相同。这是我的硬件和软件的详细信息。

  1. Tesla V100 PCIE 16GB
  2. Ubuntu 18.04
  3. NVIDIA-SMI 384.183
  4. CUDA 9.0
  5. CuDNN 7.5.0
  6. 迷你康达3
  7. Python 3.6
  8. Tensorflow 1.12
  9. Keras 2.1.6

这是我的代码。

encoder_LSTM = tf.keras.layers.CuDNNGRU(hidden_unit,return_sequences=True,return_state=True)
encoder_LSTM_rev=tf.keras.layers.CuDNNGRU(hidden_unit,return_state=True,return_sequences=True,go_backwards=True)

encoder_outputs, state_h = encoder_LSTM(x)
encoder_outputsR, state_hR = encoder_LSTM_rev(x)

这是错误消息。

2019-05-27 19:08:06.814896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-05-27 19:08:06.814956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-27 19:08:06.814971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-05-27 19:08:06.814978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-05-27 19:08:06.815279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14678 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:00:05.0, compute capability: 7.0)
2019-05-27 19:08:08.050226: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2019-05-27 19:08:08.050350: E tensorflow/stream_executor/cuda/cuda_dnn.cc:381] Possibly insufficient driver version: 384.183.0
2019-05-27 19:08:08.050378: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cudnn_rnn_ops.cc:1214 : Unknown: Fail to find the dnn implementation.
2019-05-27 19:08:08.050483: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2019-05-27 19:08:08.050523: E tensorflow/stream_executor/cuda/cuda_dnn.cc:381] Possibly insufficient driver version: 384.183.0
2019-05-27 19:08:08.050541: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cudnn_rnn_ops.cc:1214 : Unknown: Fail to find the dnn implementation.
Traceback (most recent call last):
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
     [[{{node cu_dnngru/CudnnRNN}} = CudnnRNN[T=DT_FLOAT, direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="gru", seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](cu_dnngru/transpose, cu_dnngru/ExpandDims, gradients/while/Shape/Enter_grad/zeros/Const, cu_dnngru/concat)]]
     [[{{node mean_squared_error/value/_37}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1756_mean_squared_error/value", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ta_skenario1.py", line 271, in <module>
    losss, op = sess.run([loss, optimizer], feed_dict={x:data,y_label:label,initial_input:begin_sentence})
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Fail to find the dnn implementation.
     [[node cu_dnngru/CudnnRNN (defined at ta_skenario1.py:205)  = CudnnRNN[T=DT_FLOAT, direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="gru", seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](cu_dnngru/transpose, cu_dnngru/ExpandDims, gradients/while/Shape/Enter_grad/zeros/Const, cu_dnngru/concat)]]
     [[{{node mean_squared_error/value/_37}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1756_mean_squared_error/value", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'cu_dnngru/CudnnRNN', defined at:
  File "ta_skenario1.py", line 205, in <module>
    encoder_outputs, state_h = encoder_LSTM(x)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/keras/layers/recurrent.py", line 619, in __call__
    return super(RNN, self).__call__(inputs, **kwargs)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 757, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/keras/layers/cudnn_recurrent.py", line 109, in call
    output, states = self._process_batch(inputs, initial_state)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/keras/layers/cudnn_recurrent.py", line 299, in _process_batch
    rnn_mode='gru')
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 116, in cudnn_rnn
    is_training=is_training, name=name)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/paperspace/.conda/envs/gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

UnknownError (see above for traceback): Fail to find the dnn implementation.
     [[node cu_dnngru/CudnnRNN (defined at ta_skenario1.py:205)  = CudnnRNN[T=DT_FLOAT, direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="gru", seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](cu_dnngru/transpose, cu_dnngru/ExpandDims, gradients/while/Shape/Enter_grad/zeros/Const, cu_dnngru/concat)]]
     [[{{node mean_squared_error/value/_37}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1756_mean_squared_error/value", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

有什么主意吗？谢谢

更新：我试图将CuDNN版本从7.5.0降级到7.1.4，但结果仍然相同。

4 个答案:

答案 0 :(得分：1)

这在 Tensorflow 2 中对我有用，如建议的 here

import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)

答案 1 :(得分：0)

您是否测试了安装（cuda，cudnn，tensorflow-gpu）？

测试CUDA： 首先检查是否：

$ nvcc -V

显示您的cuda工具包的正确版本。 然后，您可以通过以下过程对其进行测试：

首先（需要几分钟）：

 $ cd ~/NVIDIA_CUDA-9.0_Samples
 $ make

然后：

$ cd ~/NVIDIA_CUDA-9.0_Samples/bin/x86_64/linux/release
$./deviceQuery

如果您获得：最后的“结果：通过”，那就一切都很好！

测试Cudnn： 

$ cp -r /usr/src/cudnn_samples_v7/ $HOME
$ cd $HOME/cudnn_samples_v7/mnistCUDNN
$ make clean && make
$ ./mnistCUDNN

结果应该是：“测试通过！”

测试tensorflow-gpu：

如果cuda和cudnn正常运行，则可以使用以下命令测试tensorflow安装：

from tensorflow.python.client import device_lib
device_lib.list_local_devices()

我建议您使用以下方法在conda环境中安装tensorflow：

conda create --name tf_gpu tensorflow-gpu

对我来说（并且在遇到很多问题之后），它运行得很好。

来源： gpu installation for Ubuntu 18.04，tensorflow-gpu installation

答案 2 :(得分：0)

对于使用 TF2.0 和 Cuda 10.0 和 cuDNN-7 遇到此问题的任何人，您都可能会收到此文件，因为您不小心升级了cuDNN从7.6.2到>7.6.5。尽管TF文档指出>=7.4.1可以正常工作，但事实并非如此！降级为CudNN，如下所示：

sudo apt-get install --no-install-recommends \
  cuda-10-0 \
  libcudnn7=7.6.2.24-1+cuda10.0  \
  libcudnn7-dev=7.6.2.24-1+cuda10.0

将来，您可以通过在aptitude上标记它们来在Ubuntu / Debian中暂停对cuDNN的更新：

sudo apt-mark hold libcudnn7 libcudnn7-dev

答案 3 :(得分：0)

不确定它是否可以解决问题，但就我而言，问题是由使用多个jupyter笔记本文件引起的。

我当时正在为神经网络编写一个简单的代码，所以我决定将其分成两本笔记本，一本用于训练，一本用于预测（如果您没有资源/时间来训练您的网络，我会提供将模型保存到文件中。）

如果我“一起”运行两个笔记本，那么基本上在不断开第一个代码内核的情况下，首先进行培训，然后进行预测，那么我将得到此错误。

在使用第二台jupyter笔记本电脑之前先断开其内核可以解决我的问题。

相关问题
最新问题