Question

我试图在同一过程中并行运行两个张量流模型。

在Tensorflow 1.x中，我们可以做例如Keras Tensorflow - Exception while predicting from multiple threads

graph = tf.Graph()
with graph.as_default():
    session = tf.Session()
    with session.as_default():
        # Create and use the model ...

在Tensorflow 2.0中，该会话已被删除。

当我尝试在同一过程中并行运行两个模型时，Tensorflow崩溃。 TF 2.0是否可以解决此问题？

错误消息：

2019-09-18 16:26:56.978352: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.981418: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.983202: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.985757: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.987541: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.990140: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.992499: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.994886: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997778: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997802: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
2019-09-18 16:26:56.997829: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[{{node dense/MatMul}}]]
2019-09-18 16:26:57.000709: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:57.000744: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
  File 
    result = model.predict([numpy.array([inV,]), numpy.array([inS,])])
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 915, in predict
    use_multiprocessing=use_multiprocessing)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 722, in predict
    callbacks=callbacks)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 393, in model_iteration
    batch_outs = f(ins_batch)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py", line 3625, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1081, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1121, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError:  Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[node dense/MatMul (defined at usr/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_keras_scratch_graph_378]

Function call stack:
keras_scratch_graph

Answer 1

尝试为每种型号允许使用部分GPU。

import tensorflow as tf
config = tf.compat.v1.ConfigProto(gpu_options = 
                     tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=0.8)
# device_count = {'GPU': 1}
)
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)
tf.compat.v1.keras.backend.set_session(session)

会话和并行性在TF2.0中如何工作？

1 个答案: