Question

我正在使用tensorflow来微调模型，它使用线程将输入队列排入队列。代码在几次迭代中运行良好，但在几次迭代后仍然会出现以下错误。但是，代码通常在几次迭代后退出几个这些错误，我必须手动重启它。你能帮我理解一下这个错误。

Exception in thread Thread-29:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "data_feeder.py", line 212, in enqueue_op_online_val
    model_vars['labels_val']: labels_minibatch})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 985, in _do_call
    raise type(e)(node_def, op, message)
NotFoundError: PruneForTargets: Some target nodes not found: fifo_queue_enqueue/fifo_queue_enqueue_4/fifo_queue_enqueue_10/fifo_queue_enqueue_10/fifo_queue_enqueue_8/fifo_queue_enqueue_2/fifo_queue_enqueue_4/fifo_queue_enqueue/fifo_queue_enqueue_3/fifo_queue_enqueue_3/fifo_queue_enqueue/fifo_queue_enqueue_1/fifo_queue_enqueue_5/fifo_queue_enqueue/fifo_queue_enqueue_2/fifo_queue_enqueue/fifo_queue_enqueue_4/fifo_queue_enqueue_6/fifo_queue_enqueue_20/fifo_queue_enqueue_29/fifo_queue_enqueue/fifo_queue_enqueue_7/fifo_queue_enqueue_8/fifo_queue_enqueue_9/fifo_queue_enqueue_7/fifo_queue_enqueue/fifo_queue_enqueue_9/fifo_queue_enqueue_8/fifo_queue_enqueue_11/fifo_queue_enqueue_5/fifo_queue_enqueue_7/fifo_queue_enqueue_8/fifo_queue_enqueue_12/fifo_queue_enqueue_3/fifo_queue_enqueue_2/fifo_queue_enqueue_5/fifo_queue_enqueue_4/fifo_queue_enqueue_7/fifo_queue_enqueue_3/fifo_queue_enqueue/fifo_queue_enqueue_9/fifo_queue_enqueue_1/fifo_queue_enqueue_25/fifo_queue_enqueue_23/fifo_queue_enqueue_30/fifo_queue_enqueue_5/fifo_queue_enqueue_3/fifo_queue_enqueue_7/fifo_queue_enqueue_3/fifo_queue_enqueue/fifo_queue_enqueue_2/fifo_queue_enqueue_8/fifo_queue_enqueue_11/fifo_queue_enqueue_11/fifo_queue_enqueue/fifo_queue_enqueue_4

这里是排队操作

def enqueue_op(self, sess, model_vars, coord):
       queue = model_vars['queue_train']
       random.shuffle(self.keys)
       img_minibatch = np.zeros((self.batch_size, 224, 224, 3))
       #t = time.time()

       for indx in xrange(self.num_batches):
           labels_minibatch = []
           # Load all files in this batch
           for i,k in\
           enumerate(self.keys[indx*self.batch_size:(indx+1)*self.batch_size]):
               img = misc.imread(self.img_path + self.data[k]['filename'])
               img_minibatch[i, :, :, :] = img_proc(img)
               labels_minibatch.append(self.data[k]['labels'])

           labels_minibatch =\
           self.mlb.fit_transform(labels_minibatch).astype(float)
           sess.run([queue.enqueue([model_vars['input'],
                                    model_vars['labels']])],\
                    feed_dict={model_vars['input']: img_minibatch,
                               model_vars['labels']: labels_minibatch})

           if coord.should_stop():
               break

以下是线程创建代码段

 thr_train = []
 for i in xrange(12):              

thr_train.append(threading.Thread(target=primary_mb_feeder.enqueue_op,
                                       args=(sess, model_vars, coord)))
                thr_train[-1].setDaemon(True)
                thr_train[-1].start()

Answer 1

看起来问题是由enqueue_op()方法引起的，其中（i）在多个线程中运行，（ii）通过调用{{}为每个小批量创建新的tf.Operation 3}}。 TensorFlow不支持多个线程同时向图表添加节点，这样做效率通常很低（因为当图形不更改时，TensorFlow的工作效率最高）。

要解决此问题，我建议您重构代码，以便只调用queue.enqueue()一次，在所有输入线程之间共享返回的操作，并在每个{{1}中使用它用新数据输入的调用。

在获取数据时使用tensorflow进行排队和线程化错误

1 个答案: