Tensorflow对象检测API仅在CPU上运行时出现错误忽略设备规范/设备：GPU：0表示节点＆＃39; prefetch_queue_Dequeue＆＃39;

Question

当我开始训练时，该过程仅在CPU上发生。这是我开始时得到的信息：

  

忽略设备规格/设备：GPU：0表示节点   ＆＃39; prefetch_queue_Dequeue＆＃39;因为＆quot; prefetch_queue＆＃39;的输入边缘   是参考连接，并且已将设备字段设置为   /装置：CPU：0

虽然GPU内存已满，但只有突然的gpu负载峰值，而且大多数是0％。不用说，这种表现极其缓慢

似乎所有内容都加载到CPU而不是GPU，因为CPU利用率接近100％：

我也注意到在ubuntu中存在相同的问题，但它至少比windows分支快4倍（每个步骤需要400毫秒，而在Windows中需要1300毫秒）
我在Ubuntu（14.04）和Windows上都使用了tensorflow 1.3.0，它们都是使用pip install --upgrade tensorflow-gpu command安装的。 这是整个日志：

G:\Tensorflow_section\models-master\object_detection>python train.py  --logtostderr --train_dir=training_stuff --pipeline_config_path=ssd_mobilenet_v1_pets.config
INFO:tensorflow:Summary name Learning Rate is illegal; using Learning_Rate instead.
WARNING:tensorflow:From C:\Users\Master\Anaconda3\envs\anaconda35\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\meta_architectures\ssd_meta_arch.py:607: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Please use tf.global_variables instead.
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
2017-09-18 03:44:08.545358: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-09-18 03:44:08.545474: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-18 03:44:09.121357: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.835
pciBusID 0000:01:00.0
Total memory: 8.00GiB
Free memory: 6.63GiB
2017-09-18 03:44:09.121483: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-09-18 03:44:09.122196: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-09-18 03:44:09.133158: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
INFO:tensorflow:Restoring parameters from training_stuff\model.ckpt-0
2017-09-18 03:44:15.528390: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\simple_placer.cc:697] Ignoring device specification /device:GPU:0 for node 'prefetch_queue_Dequeue' because the input edge from 'prefetch_queue' is a reference connection and already has a device field set to /device:CPU:0
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path training_stuff\model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:global step 1: loss = 20.1465 (18.034 sec/step)
INFO:tensorflow:global step 2: loss = 15.8647 (1.601 sec/step)
INFO:tensorflow:global step 3: loss = 13.3987 (1.540 sec/step)
INFO:tensorflow:global step 4: loss = 11.5424 (1.562 sec/step)
INFO:tensorflow:global step 5: loss = 10.8328 (1.337 sec/step)
INFO:tensorflow:global step 6: loss = 10.7179 (1.317 sec/step)
INFO:tensorflow:global step 7: loss = 9.7616 (1.369 sec/step)
INFO:tensorflow:global step 8: loss = 8.5631 (1.336 sec/step)
INFO:tensorflow:global step 9: loss = 7.2683 (1.384 sec/step)

有什么问题以及如何解决这个问题？ 顺便说一下，这里有一些补充信息：

• 操作系统平台和分发：Windows 10 x64 1703, Build 15063.540
• 从（源代码或二进制代码）安装的TensorFlow：binary (used pip install )
• TensorFlow版本（使用下面的命令）：1.3.0
• Python版本：3.5.3
• CUDA / cuDNN版本：Cuda 8.0 /cudnn v6.0
• GPU型号和内存：GTX-1080 - 8G

更新：
评论部分中建议的代码段的输出如下：

import tensorflow as tf

sess = tf.InteractiveSession()
with tf.device('/cpu:0'):
    q = tf.FIFOQueue(3, 'float')
    add_op = q.enqueue(42)
with tf.device('/gpu:0'):
    get_op = q.dequeue()

sess.run(add_op)
print( sess.run(get_op))

输出：

2017-09-17 23:30:59.539728: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-09-17 23:30:59.539857: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-17 23:30:59.856904: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.835
pciBusID 0000:01:00.0
Total memory: 8.00GiB
Free memory: 6.63GiB
2017-09-17 23:30:59.857034: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-09-17 23:30:59.858320: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-09-17 23:30:59.858688: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
2017-09-17 23:30:59.879245: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\simple_placer.cc:697] Ignoring device specification /device:GPU:0 for node 'fifo_queue_Dequeue' because the input edge from 'fifo_queue' is a reference connection and already has a device field set to /job:localhost/replica:0/task:0/device:CPU:0
42.0

Answer 1

好的，我也在Ubuntu上进行了一些测试，这是我发现的： 计算在GPU Ubuntu上运行并与使用pip安装的Windows（Tensorflow 1.3）进行比较，速度提高4到6倍。
它利用GPU和CPU并且CPU内核上的负载平衡似乎是相当分散的。但是，Windows版本不是这种情况。
在Windows下，它似乎确实在使用GPU，因为它占用了{1}}的7.1千兆字节，但VRAM上的负载却很疯狂！与GPU上的负载微不足道的是，所有内核都被最大化并且它会扼杀整个系统的性能 进一步的调查让我想到了这个类似的问题：Object detection using GPU on Windows is about 5 times slower than on Ubuntu这个人说这是因为Windows上存在一些依赖性问题，Tensorflow自1.2版以来就没有做任何事情！ 所以最重要的是，如果您打算使用CPU，并打算有效利用您的Object Detection API和资源，请使用GPU并在那里进行实验。 Ubuntu Tensorflow似乎没有对Windows发表任何看法！