Keras生成器在一个时期后给出错误

时间:2018-06-21 08:08:08

标签: tensorflow keras pre-trained-model

我定义了一个生成器,该生成器根据预先训练的基本模型(resnet或vgg16或vgg19或...)来计算特征。

然后使用这些功能作为输入

来训练我的顶级模特
model.fit_generator(generator=features_train_generator,
                               validation_data=features_val_generator,
                               epochs=args['epochs'],
                               use_multiprocessing=False,
                               verbose=1)

培训可以开始,但是经过一个时期后,我遇到了一个很大的错误(请参阅下文)。有人知道这里发生了什么吗?显然,应该初始化的东西不是,但是我不明白。

当我在本地运行它(Keras 2.1.6和tf 1.8.0)时,一切正常,但是当我在Azure虚拟机(Keras 2.1.4和tf 1.5.0)上运行时,我得到了错误,所以它取决于版本。

File "C:\Anaconda\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_call
return fn(*args)
File "C:\Anaconda\lib\site-packages\tensorflow\python\client\session.py", line 1329, in _run_fn
status, run_metadata)
File "C:\Anaconda\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: FetchOutputs node IsVariableInitialized_535/IsVariableInitialized_4/IsVariableInitialized_48:0: not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Anaconda\lib\site-packages\keras\utils\data_utils.py", line 564, in get
inputs = self.queue.get(block=True).get()
File "C:\Anaconda\lib\multiprocessing\pool.py", line 644, in get
raise self._value
File "C:\Anaconda\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\Anaconda\lib\site-packages\keras\utils\data_utils.py", line 390, in get_index
return _SHARED_SEQUENCES[uid][i]
File "C:\SDC\src\generators.py", line 117, in __getitem__
features, y = self.__data_generation(list_IDs_temp)
File "C:\SDC\src\generators.py", line 163, in __data_generation
features = mu.get_features(X, self.base_model)
File "C:\SDC\src\model_utils.py", line 160, in get_features
features = pre_trained_model.predict(X)
File "C:\Anaconda\lib\site-packages\keras\engine\training.py", line 1842, in predict
verbose=verbose, steps=steps)
File "C:\Anaconda\lib\site-packages\keras\engine\training.py", line 1337, in _predict_loop
batch_outs = f(ins_batch)
File "C:\Anaconda\lib\site-packages\keras\backend\tensorflow_backend.py", line 2473, in __call__
session = get_session()
File "C:\Anaconda\lib\site-packages\keras\backend\tensorflow_backend.py", line 189, in get_session
[tf.is_variable_initialized(v) for v in candidate_vars])
File "C:\Anaconda\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "C:\Anaconda\lib\site-packages\tensorflow\python\client\session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Anaconda\lib\site-packages\tensorflow\python\client\session.py", line 1344, in _do_run
options, run_metadata)
File "C:\Anaconda\lib\site-packages\tensorflow\python\client\session.py", line 1363, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: FetchOutputs node IsVariableInitialized_535/IsVariableInitialized_4/IsVariableInitialized_48:0: not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "train.py", line 302, in <module>
main()
File "train.py", line 169, in main
verbose=1)
File "C:\Anaconda\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Anaconda\lib\site-packages\keras\models.py", line 1253, in fit_generator
initial_epoch=initial_epoch)
File "C:\Anaconda\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Anaconda\lib\site-packages\keras\engine\training.py", line 2262, in fit_generator
workers=0)
File "C:\Anaconda\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Anaconda\lib\site-packages\keras\engine\training.py", line 2377, in evaluate_generator
generator_output = next(output_generator)
File "C:\Anaconda\lib\site-packages\keras\utils\data_utils.py", line 570, in get
six.raise_from(StopIteration(e), e)
File "<string>", line 3, in raise_from
StopIteration: FetchOutputs node      IsVariableInitialized_535/IsVariableInitialized_4/IsVariableInitialized_48:0: not found

0 个答案:

没有答案