Tensorflow CIFAR10每走一步都会减慢

Question

系统信息

• 操作系统：Linux Ubuntu 16.04
• 从安装的TensorFlow：已尝试二进制和源
• TensorFlow版本：1.4.0-19-ga52c8d9,1.4.1
• Python版：2.7.12
• CUDA / cuDNN版本：8.0.61
• 硬件：GPU：NVIDIA GeForce GTX 1080 Ti（11GB），内存：64GB，CPU：Intel i7-6850K
• 重现的确切命令：python cifar10_train.py

描述问题

CIFAR10 Tensorflow tutorial似乎有一些奇怪的模式，它涉及它可以计算的每秒例子数：   - 第0步非常慢   - 每100步都非常慢   - 在一个非常缓慢的步骤之后的步骤要么有点慢，要么平均   - 接下来的10-30步骤略有提升（快于平均水平）   - 其余步骤是平均速度

我希望（按重要性顺序）：   - 解释和修复每100步这么慢   - 解释和说明，告诉我如何使每一步都以提升的速度运行（慢速后不久的速度）   - 对慢速0和第1步的解释和修复

我找不到每100步发生的任何其他日志记录或处理。可能是tf.train.MonitoredSession吗？

重现性：

• 在CPU而非GPU上进行培训时
• 与批量大小无关
• 在MacBook Pro上（Retina，13英寸，2014年中）

硬件利用率：

1. 平均：

   • CPU：82-84％
   • GPU：70-85％
   • RAM：3.7GB

2. 每100步：

   • CPU：9％
   • GPU：0％
   • RAM：3.7GB

3. 缓慢步骤后提升：

   • GPU：92％
   • CPU：82-84％
   • RAM：3.7GB

4. 空闲：

   • CPU：1％
   • GPU：0％
   • RAM：1.6GB

整体CPU和RAM使用率（清楚地显示CPU每100步）：

日志摘录(full logs)：

step 0: (587.3 examples/sec; 6.974 sec/batch)
step 1: (22630.6 examples/sec; 0.181 sec/batch)
step 2: (36253.6 examples/sec; 0.113 sec/batch)
step 3: (37966.0 examples/sec; 0.108 sec/batch)
step 4: (38511.4 examples/sec; 0.106 sec/batch)
step 5: (38554.6 examples/sec; 0.106 sec/batch)
step 6: (32112.4 examples/sec; 0.128 sec/batch)
step 7: (38912.4 examples/sec; 0.105 sec/batch)
step 8: (39377.0 examples/sec; 0.104 sec/batch)
step 9: (38206.2 examples/sec; 0.107 sec/batch)
step 10: (38222.1 examples/sec; 0.107 sec/batch)
step 11: (38757.5 examples/sec; 0.106 sec/batch)
step 12: (38833.1 examples/sec; 0.105 sec/batch)
step 13: (39774.8 examples/sec; 0.103 sec/batch)
step 14: (39795.9 examples/sec; 0.103 sec/batch)
step 15: (37850.5 examples/sec; 0.108 sec/batch)
step 16: (38443.5 examples/sec; 0.107 sec/batch)
step 17: (39194.6 examples/sec; 0.105 sec/batch)
step 18: (39164.0 examples/sec; 0.105 sec/batch)
step 19: (39057.5 examples/sec; 0.105 sec/batch)
step 20: (33268.7 examples/sec; 0.123 sec/batch)
step 21: (39459.7 examples/sec; 0.104 sec/batch)
step 22: (39336.2 examples/sec; 0.104 sec/batch)
step 23: (39207.1 examples/sec; 0.104 sec/batch)
step 24: (39330.5 examples/sec; 0.104 sec/batch)
step 25: (38783.9 examples/sec; 0.106 sec/batch)
step 26: (39038.9 examples/sec; 0.105 sec/batch)
step 27: (39214.2 examples/sec; 0.104 sec/batch)
step 28: (39525.9 examples/sec; 0.104 sec/batch)
step 29: (37209.0 examples/sec; 0.110 sec/batch)
step 30: (38356.7 examples/sec; 0.107 sec/batch)
step 31: (36077.0 examples/sec; 0.114 sec/batch)
step 32: (37143.8 examples/sec; 0.110 sec/batch)
step 33: (35961.1 examples/sec; 0.114 sec/batch)
step 34: (33378.4 examples/sec; 0.123 sec/batch)
step 35: (37830.3 examples/sec; 0.108 sec/batch)
step 36: (36789.5 examples/sec; 0.111 sec/batch)
step 37: (36638.2 examples/sec; 0.112 sec/batch)
step 38: (36848.1 examples/sec; 0.111 sec/batch)
step 39: (36041.4 examples/sec; 0.114 sec/batch)
step 40: (36612.0 examples/sec; 0.112 sec/batch)
step 41: (35623.9 examples/sec; 0.115 sec/batch)
step 42: (37589.3 examples/sec; 0.109 sec/batch)
step 43: (37462.9 examples/sec; 0.109 sec/batch)
step 44: (35823.6 examples/sec; 0.114 sec/batch)
step 45: (35911.8 examples/sec; 0.114 sec/batch)
step 46: (36073.8 examples/sec; 0.114 sec/batch)
step 47: (36930.2 examples/sec; 0.111 sec/batch)
step 48: (36142.9 examples/sec; 0.113 sec/batch)
...
step 99: (36434.8 examples/sec; 0.112 sec/batch)
step 100: (1215.0 examples/sec; 3.371 sec/batch)
step 101: (35952.9 examples/sec; 0.114 sec/batch)
step 102: (38422.5 examples/sec; 0.107 sec/batch)
step 103: (39315.8 examples/sec; 0.104 sec/batch)
step 104: (38989.1 examples/sec; 0.105 sec/batch)
step 105: (39091.4 examples/sec; 0.105 sec/batch)
step 106: (39247.6 examples/sec; 0.104 sec/batch)
step 107: (38009.7 examples/sec; 0.108 sec/batch)
step 108: (38746.7 examples/sec; 0.106 sec/batch)
step 109: (39505.4 examples/sec; 0.104 sec/batch)
step 110: (39340.0 examples/sec; 0.104 sec/batch)
step 111: (39065.0 examples/sec; 0.105 sec/batch)
step 112: (38561.1 examples/sec; 0.106 sec/batch)
step 113: (39109.0 examples/sec; 0.105 sec/batch)
step 114: (39203.7 examples/sec; 0.104 sec/batch)
step 115: (39144.4 examples/sec; 0.105 sec/batch)
step 116: (38317.6 examples/sec; 0.107 sec/batch)
step 117: (33757.5 examples/sec; 0.121 sec/batch)
step 118: (34115.4 examples/sec; 0.120 sec/batch)
step 119: (35671.4 examples/sec; 0.115 sec/batch)
step 120: (35297.2 examples/sec; 0.116 sec/batch)
step 121: (36152.8 examples/sec; 0.113 sec/batch)
step 122: (35780.1 examples/sec; 0.114 sec/batch)
step 123: (35847.1 examples/sec; 0.114 sec/batch)
step 124: (36888.9 examples/sec; 0.111 sec/batch)
step 125: (36946.2 examples/sec; 0.111 sec/batch)
...