运行代码时,每次都会收到此消息:
2018-09-27 19:31:03.353933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: GeForce GTX 650 Ti major: 3 minor: 0 memoryClockRate(GHz): 0.941
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.65GiB
2018-09-27 19:31:03.355743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-27 19:31:04.822514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-27 19:31:04.822895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-09-27 19:31:04.823072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-09-27 19:31:04.823679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1416 MB memory) -> physical GPU (device: 0, name: GeForce GTX 650 Ti, pci bus id: 0000:01:00.0, compute capability: 3.0)
2018-09-27 19:31:12.050251: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 261.79MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-09-27 19:31:17.191146: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
最后两条消息是警告,对我来说似乎很奇怪:我应该有1.65GiB的可用内存,但不能分配一些较小的数量。 我该如何解决?该消息的来源是什么?而且:为什么我的GPU使用率不能超过50%?
这是我开始训练时的样子:
代码本身在my repo中(很难知道我的代码的哪些部分是相关的)。
答案 0 :(得分:0)
您似乎没有使用多GPU模型?
答案 1 :(得分:0)
您是否尝试过增加批量大小?我从您的代码中看到您使用了add-customer