我正在使用Keras，Tensorflow GPU后端和CUDA 9.1构建图像分类系统，在Ubuntu 18.04上运行。
我使用的是非常大的图像数据集，包含120万张图像，15k类，并且大小为335 GB。
我可以毫无问题地训练我的网络上90,000张图像。但是，当我扩展并使用120万个图像的整个数据集时，我得到下面显示的错误，我认为这与内存不足有关。
我使用的是带有11GB内存的GeForce GTX 1080，我有128GB的RAM，300GB的交换文件和16个内核的AMD Threadripper 1950X。
我遵循了解决类似问题的建议。我现在使用的是smaller batch size of 10或even smaller，以及smaller dense inner layer的256，我在第一个训练纪元开始之前仍然会收到如下所示的相同错误。< / p>
[更新]：我发现在VGG16 predict_generator通话期间发生内存错误，甚至在我的网络建立或训练之前。请参阅下面的代码。
首先，警告和错误：
2018-05-19 20:24:01.255788: E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 5635855360 bytes on host: CUresult(304)
2018-05-19 20:24:01.255850: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 5635855360
然后例外：
2018-05-19 13:56:40.472404: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 68719476736
InUse: 15548829696
MaxInUse: 15548829696
NumAllocs: 15542
MaxAllocSize: 16777216
2018-05-19 13:56:40.472563: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****************************************************************************************************
Traceback (most recent call last):
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: block5_pool/MaxPool/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_133_block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bottleneck.py", line 37, in <module>
bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/engine/training.py", line 2510, in predict_generator
outs = self.predict_on_batch(x)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/engine/training.py", line 1945, in predict_on_batch
outputs = self.predict_function(ins)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2478, in __call__
**self.session_kwargs)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: block5_pool/MaxPool/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_133_block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
这是我的代码：
import numpy as np
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.layers import Dropout, Flatten, Dense
from keras.models import Sequential
from keras.preprocessing.image import ImageDataGenerator
from keras import applications
from keras.utils.np_utils import to_categorical
import matplotlib.pyplot as plt
# Dimensions of our images.
img_width, img_height = 224, 224
train_data_dir = './train_sample'
epochs = 100
batch_size = 10
# Data preprocessing
# Pixel values rescaling from [0, 255] to [0, 1] interval
datagen = ImageDataGenerator(rescale=1. / 255)
# Retrieve images and their classes for training set.
train_generator_bottleneck = datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
class_mode=None,
shuffle=False)
num_classes = len(train_generator_bottleneck.class_indices)
model_vgg = applications.VGG16(include_top=False, weights='imagenet')
bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck)
np.save('../models/bottleneck_features_train.npy', bottleneck_features_train)
train_data = np.load('../models/bottleneck_features_train.npy')
train_labels = to_categorical(train_generator_bottleneck.classes, num_classes=num_classes)
model_top = Sequential()
model_top.add(Flatten(input_shape=train_data.shape[1:]))
model_top.add(Dense(256, activation='relu'))
model_top.add(Dropout(0.5))
model_top.add(Dense(num_classes, activation='softmax'))
model_top.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
# Model saving callback
checkpointer = ModelCheckpoint(filepath='../models/bottleneck_features.h5', monitor='val_acc', verbose=1,
save_best_only=True)
# Early stopping
early_stopping = EarlyStopping(monitor='val_acc', verbose=1, patience=5)
history = model_top.fit(
train_data,
train_labels,
verbose=2,
epochs=epochs,
batch_size=batch_size,
callbacks=[checkpointer, early_stopping],
validation_split=0.3)
答案 0 :(得分：1)
我不相信这里的问题是batch_size，因为你提到它已经很低了。此外，因为你说它适用于90k图像，问题可能是train_data无法适应内存中的GPU（在每个拟合时期开始时需要）。为了缓解此问题，您需要使model_top符合generator，就像从predict_generator获取功能一样。你可以做到这一点的一种方法是在train_data周围包装一个生成器类，但我只想连接这两个模型（注意我无法测试这个，但我认为它是正确的）：
model_vgg = applications.VGG16(include_top=False, weights='imagenet')
model_top = Flatten()(model_vgg)
model_top = Dense(256, activation='relu')(model_top)
model_top = Dropout(0.3)(model_top)
model_top = Dense(num_classes, activation='softmax')(model_top)
model = Model(inputs=model_vgg.inputs, outputs=model_top)
model.compile(loss='sparse_categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
# Model saving callback
checkpointer = ModelCheckpoint(filepath='../models/bottleneck_features.h5', monitor='val_acc', verbose=1,
save_best_only=True)
# Early stopping
early_stopping = EarlyStopping(monitor='val_acc', verbose=1, patience=5)
history = model.fit_generator(
train_data,
train_labels,
verbose=2,
steps_per_epoch=steps_per_epoch,
batch_size=batch_size,
callbacks=[checkpointer, early_stopping],
...)
我将categorical_crossentropy更改为sparse_categorical_crossentropy，以便只能将索引作为标签发送，否则相同。您需要提供steps_per_epoch作为训练数据的长度/批量大小。或者只是输入任何数字进行测试。我还使用了keras功能api来使其更清晰。
这也可以让VGG top的重量发生变化，以帮助您更好地进行分类。如果由于某种原因这不是您想要的，您可以通过迭代所有vgg图层并将trainable设置为false来冻结它。
lmk如果有效。