进入" PoolAllocator"需很长时间

时间:2016-04-29 02:07:08

标签: tensorflow

您好我对poolallocator有疑问。当我开始我的培训工作时,花了几个小时试图做#34; PoolAllocator"。一些日志如下所示。有没有办法调试/分析原因?我怎样才能改进它? 谢谢!

tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 9639 get requests, put_count=4341 evicted_count=1000 eviction_rate=0.230362 and unsatisfied allocation rate=0.663762
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 100 to 110
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2013 evicted_count=2000 eviction_rate=0.993542 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 7080 get requests, put_count=6922 evicted_count=5000 eviction_rate=0.722335 and unsatisfied allocation rate=0.730791
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 176 to 193
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2025 evicted_count=2000 eviction_rate=0.987654 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=5030 evicted_count=5000 eviction_rate=0.994036 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2044 evicted_count=2000 eviction_rate=0.978474 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 9617 get requests, put_count=8892 evicted_count=5000 eviction_rate=0.562303 and unsatisfied allocation rate=0.600915
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 596 to 655


I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2087 evicted_count=2000 eviction_rate=0.958313 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=3095 evicted_count=3000 eviction_rate=0.969305 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1115 evicted_count=1000 eviction_rate=0.896861 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1140 evicted_count=1000 eviction_rate=0.877193 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1169 evicted_count=1000 eviction_rate=0.855432 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1204 evicted_count=1000 eviction_rate=0.830565 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2247 evicted_count=2000 eviction_rate=0.890076 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=8272 evicted_count=8000 eviction_rate=0.967118 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2362 evicted_count=2000 eviction_rate=0.84674 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 10614 get requests, put_count=10944 evicted_count=2000 eviction_rate=0.182749 and unsatisfied allocation rate=0.198606
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 4823 to 5305
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=3705 evicted_count=3000 eviction_rate=0.809717 and unsatisfied allocation rate=0
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 4204990 get requests, put_count=4204742 evicted_count=3000 eviction_rate=0.00071348 and unsatisfied allocation rate=0.00104257
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 16377314 get requests, put_count=16374197 evicted_count=13000 eviction_rate=0.000793932 and unsatisfied allocation rate=0.00105347

2 个答案:

答案 0 :(得分:1)

实际上工作正常。问题是python的输出缓冲。在工作完成之前,它不会显示任何培训结果。您可以通过以下方式禁用缓冲:

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) 

或者您可以尝试其他方法: Disable output buffering

答案 1 :(得分:1)

工作正常。如果内存不足,这些消息应该引起关注。

这个问题的答案可能对你有所帮助。 How to interpret Poolallocator messages in tensorflow?