无法在PyAgender上使用GPU创建cudnn句柄:CUDNN_STATUS_INTERNAL_ERROR

时间:2019-01-03 00:54:45

标签: python tensorflow

示例代码在CUDNN上运行良好,但使用通过pip返回安装的代码无法创建cudnn句柄:CUDNN_STATUS_INTERNAL_ERROR。

我正在使用通过GPU 2080(x2)编译的Tensorflow 1.12,CUDA 10.0和CUDNN 7.4.1

这是Ubuntu 16.04 LTS。我也使用venv。一切都使用Python 3.5.2。在venv中完成。

我正在尝试使用https://pypi.org/project/py-agender/中的Py-Agender

使用GPU标志编译CUDA并成功验证它是否有效后,我尝试: enter code here pip3安装py-agender。

运行它会导致内部CUDNN错误。

我已按照其他回复的建议删除了〜/ .nv,但没有帮助

我还确认了CuDNN实际上可以使用GPU:

print(device_lib.list_local_devices()) returns:

2019-01-03 09:31:30.353768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-01-03 09:31:30.354169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0

...所以我们至少有一些可行的方法!

b。 mNIST样本运行并返回预期值。

代码部分如下:

agender = PyAgender() 

def get_gender(value) :
    if value > 0.5:
        return 'female'
    else :
        return 'male'

input_dir = sys.argv[1]

directories = os.listdir(input_dir)

if len(directories):
    for video in directories:
        files = glob.glob(input_dir + video + '/*')

        if len(files):
            for file_path in files:

                filename, file_extension = os.path.splitext(os.path.basename(file_path))
                faces = agender.detect_genders_ages(cv2.imread(file_path))

该代码仅在使用CPU编译的Tensorflow上运行良好。使用GPU,完整的错误是:

$ python age-gender.py ./frames/part1/

Using TensorFlow backend.
- Began to analyze the age and gender 
2019-01-03 09:44:24.229618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-01-03 09:44:24.230019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0 totalMemory: 7.76GiB freeMemory: 7.62GiB
2019-01-03 09:44:24.302145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] su
ccessful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-01-03 09:44:24.302602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found
 device 1 with properties:name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:02:00.0 totalMemory: 7.77GiB freeMemory: 7.62GiB
2019-01-03 09:44:24.302625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Addin
g visible gpu devices: 0, 1
2019-01-03 09:44:24.665465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device
 interconnect StreamExecutor with strength 1 edge matrix:
2019-01-03 09:44:24.665492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1
2019-01-03 09:44:24.665499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y
2019-01-03 09:44:24.665504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N
2019-01-03 09:44:24.665749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7335 MB memory) ->
 physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-01-03 09:44:24.666056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7338 MB memory) ->
 physical GPU (device: 1, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
2019-01-03 09:44:25S :: Filename: ./frames/part1/2017/658.jpg
2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/698.jpg
2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/746.jpg 
2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/371.jpg 
2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/218.jpg 
2019-01-03 09:44:25S :: Filename:  ./frames/part1/2017/860.jpg 
2019-01-03 09:44:26.702699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-01-03 09:44:26.709821: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
  File "age-gender.py", line 50, in <module>
    faces = agender.detect_genders_ages(cv2.imread(file_path))
  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 56, in detect_genders_ages
    height=face['height'])
  File "/home/sv/.local/lib/python3.5/site-packages/pyagender/pyagender.py", line 82, in gender_age
    result = self.resnet.predict(np.array([test_img]))
  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1169, in predict
    steps=steps)
  File "/opt/inta/venv/lib/python3.5/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "/opt/inta/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
    run_metadata_ptr)
  File "/opt/inta/venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_1/cond/FusedBatchNorm/Switch"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_1/convolution-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]

0 个答案:

没有答案