Keras内存不足的后果

时间:2019-01-02 10:47:04

标签: tensorflow keras out-of-memory gpu

如果这个问题不在这里,请随时参考另一个StackExchange网站。 :-)

我正在使用Keras,并且我的GPU(GeForce GTX 970,〜4G)上的内存非常有限。因此,结果是我的内存不足(OOM)与Keras一起使用,批处理大小设置为高于一定水平。降低批次大小我没有这个问题,但是Keras输出以下警告:

2019-01-02 09:47:03.173259: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.57GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.211139: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.68GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.268074: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.95GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.685032: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.39GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.732304: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.56GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.850711: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.39GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.879135: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.48GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.963522: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.42GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:03.984897: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.47GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-01-02 09:47:04.058733: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.08GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

这些警告对我作为用户意味着什么?这些性能提升是什么?这是否意味着它只是计算速度更快,还是就更好的验证损失而言,我甚至可以获得更好的结果?

在我的设置中,我将Keras与Tensorflow后端一起使用,并且tensorflow-gpu == 1.8.0。

1 个答案:

答案 0 :(得分:1)

这意味着,由于GPU无法用于某些操作,因此培训会在速度方面损失一些效率。但是,损失的结果不应受到影响。

为避免此问题,最佳实践是减小批处理大小以有效利用可用的GPU内存。