预测时出现问题。早先它起作用,并且在conda环境中没有任何更改。由于以下原因,我不确定这是CUDA问题:
tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
得到:UbuntU 18.04,tf-gpu == 1.13.1 CUDA DRVIER VERSION 10.0 GPUGeForce RTX 2080 Ti
输入的批处理也不为空。
不知道该怎么办。在完整的错误日志下方:
tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-08-08 10:59:46.010315: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at cudnn_rnn_ops.cc:1224 : Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1225): 'cudnnSetTensorNdDescriptor( tensor_desc.get(), data_type, sizeof(dims) / sizeof(dims[0]), dims, strides)'
2019-08-08 10:59:46.010454: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at cudnn_rnn_ops.cc:1224 : Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1225): 'cudnnSetTensorNdDescriptor( tensor_desc.get(), data_type, sizeof(dims) / sizeof(dims[0]), dims, strides)'
Traceback (most recent call last):
File "model_test.py", line 61, in <module>
input_term = deepspeech([row['path']])[0]
File "/home/ai/dp_keras_model_cnn_extension/DeepSpeech-Keras/source/deepspeech.py", line 68, in __call__
y_hat = self.predict(X)
File "/home/ai/dp_keras_model_cnn_extension/DeepSpeech-Keras/source/deepspeech.py", line 104, in predict
return self.distributed_model.predict_on_batch(X)
File "/home/ai/anaconda3/envs/voice-mvp/lib/python3.6/site-packages/keras/engine/training.py", line 1274, in predict_on_batch
outputs = self.predict_function(ins)
File "/home/ai/anaconda3/envs/voice-mvp/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "/home/ai/anaconda3/envs/voice-mvp/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "/home/ai/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
run_metadata_ptr)
File "/home/ai/.local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1225): 'cudnnSetTensorNdDescriptor( tensor_desc.get(), data_type, sizeof(dims) / sizeof(dims[0]), dims, strides)'
[[{{node replica_0/DeepSpeech/bidirectional_1/CudnnRNN_1}}]]
[[{{node replica_1/DeepSpeech/time_distributed_2/Reshape_1}}]]