我已经创建了tfrecords作为我的数据库。该数据库由9个不同的tfrecord文件组成。目的是将来自9个数据库的一批样品输入模型。因此，我使用了TFRecordDataset的zip功能。每个样本都包含一个框架及其功能集。因此，我需要从每个tfrecord文件中取出8个样本，这样一共可以得到72个(features, image)。因此，我单独使用图像提取特征，如下面的代码所示。
问题：当我到达第1纪元的末尾时，剩余的数据小于72.结果，来自第二纪元的数据被添加到一批72个样本中。所以，由于我正在训练一个递归神经网络，所以这在我的情况下是不可定的，所以我有一个应该是一致的状态（现在没有必要讨论）。
因此，我没有使用repeat函数，而是尝试实现https://www.tensorflow.org/programmers_guide/datasets Under Processing multiple epochs中提到的内容，即使用try和except循环。
# Compute for 100 epochs.
for _ in range(100):
sess.run(iterator.initializer)
while True:
try:
sess.run(next_element)
except tf.errors.OutOfRangeError:
break
# [Perform end-of-epoch calculations here.]
一旦我这样做，我就陷入了另一个问题。首先是我的完整代码：
import tensorflow as tf
import numpy as np
import time
import cv2
num_epoch = 1
batch_size = 8 # This is set to 8 since
num_threads = 9
common = "C:/Users/user/PycharmProjects/AffectiveComputingNew/database/"
filenames = [(common + "train_1_db.tfrecords"), (common + "train_2_db.tfrecords"), (common + "train_3_db.tfrecords"),
(common + "train_4_db.tfrecords"), (common + "train_5_db.tfrecords"), (common + "train_6_db.tfrecords"),
(common + "train_7_db.tfrecords"), (common + "train_8_db.tfrecords"), (common + "train_9_db.tfrecords")]
# Transforms a scalar string `example_proto` into a pair of a scalar string and
# a scalar integer, representing an image and its label, respectively.
def _parse_function(example_proto):
features = {
'height': tf.FixedLenFeature([], tf.int64),
'width': tf.FixedLenFeature([], tf.int64),
'image_raw': tf.FixedLenFeature([], tf.string),
'features': tf.FixedLenFeature([432], tf.float32)
}
parsed_features = tf.parse_single_example(example_proto, features)
# This is how we create one example, that is, extract one example from the database.
image = tf.decode_raw(parsed_features['image_raw'], tf.uint8)
# The height and the weights are used to
height = tf.cast(parsed_features['height'], tf.int32)
width = tf.cast(parsed_features['width'], tf.int32)
# The image is reshaped since when stored as a binary format, it is flattened. Therefore, we need the
# height and the weight to restore the original image back.
image = tf.reshape(image, [height, width, 3])
features = parsed_features['features']
return features, image
random_features = tf.Variable(tf.zeros([72, 432], tf.float32))
random_images = tf.Variable(tf.zeros([72, 112, 112, 3]))
datasets = []
for _ in filenames:
datasets.append(tf.contrib.data.TFRecordDataset(_).map(_parse_function))
dataset_ziped = tf.contrib.data.TFRecordDataset.zip((datasets[0], datasets[1], datasets[2], datasets[3],
datasets[4], datasets[5], datasets[6], datasets[7], datasets[8]))
#dataset = dataset_ziped.repeat(num_epoch)
dataset = dataset_ziped.batch(batch_size)
iterator = dataset.make_initializable_iterator()
next_batch = iterator.get_next() # This has shape: [9, 2]
features = tf.concat((next_batch[0][0], next_batch[1][0], next_batch[2][0], next_batch[3][0],
next_batch[4][0], next_batch[5][0], next_batch[6][0], next_batch[7][0],
next_batch[8][0]), axis=0)
features = tf.reshape(features, shape=[9, 8, 432]) # where 8 * 9 = 72
features = tf.transpose(features, perm=[1, 0, 2]) # shape becomes: [8, 9, 432]
features = tf.reshape(features, shape=[72, 432]) # Now frames will be: 1st frame from 1st video, second from second video...
images = tf.concat((next_batch[0][1], next_batch[1][1], next_batch[2][1], next_batch[3][1],
next_batch[4][1], next_batch[5][1], next_batch[6][1], next_batch[7][1],
next_batch[8][1]), axis=0)
images = tf.reshape(images, shape=[9, 8, 112, 112, 3])
images = tf.transpose(images, perm=[1, 0, 2, 3, 4])
images = tf.reshape(images, shape=[72, 112, 112, 3])
init_op = tf.global_variables_initializer()
with tf.Session() as sess:
# Initialize `iterator` with training data.
sess.run(init_op)
for _ in range(num_epoch):
sess.run(iterator.initializer)
# This while loop will run indefinitly until the end of the first epoch
while True:
try:
lst = []
features_np = sess.run([features])[0] # since the output is always: (1, 72, 432)
for f in features_np:
lst.append(f[0])
except tf.errors.OutOfRangeError:
print('errorrrrr')
因此，由于样本数不再是72，我在第一行遇到错误：
features = tf.reshape(features, shape=[9, 8, 432]) # where 8 * 9 = 72
所以，我需要一种方法来处理这个错误。我尝试断言如下：
assert_op = tf.Assert(tf.equal(tf.shape(features[0]), batch_size * 9), [features])
with tf.control_dependencies([assert_op])... after features = tf.concat...
它不起作用。我按照以下方式尝试了tf.cond（并且它也不起作用）：
tf.cond(tf.equal(tf.shape(features)[0], batch_size * 9),
lambda: tf.assign(random_features, features),
lambda: tf.assign(random_features, random_features))
features = tf.reshape(random_features, shape=[9, 8, 432]) # where 8 * 9 = 72
....
总之，我需要迭代时代而不交错来自不同迭代的样本，同时在使用reshape函数时保持特征问题（批量大小小于72）在我的情况下）。
非常感谢任何帮助!!
答案 0 :(得分：0)
因此，由于最后一批被截断，我创建了一个临时variable，只要批量大小等于代码中指定的batch_size，就会为其分配初始批次的值。因此，一旦当前批次的大小不再等于batch_size，我就使用了我创建的临时批处理。以下是以下解决方案：
import tensorflow as tf
import numpy as np
import time
import cv2
num_epoch = 2
batch_size = 8 # This is set to 8 since
num_threads = 9
common = "C:/Users/user/PycharmProjects/AffectiveComputingNew/database/"
filenames = [(common + "train_1_db.tfrecords"), (common + "train_2_db.tfrecords"), (common + "train_3_db.tfrecords"),
(common + "train_4_db.tfrecords"), (common + "train_5_db.tfrecords"), (common + "train_6_db.tfrecords"),
(common + "train_7_db.tfrecords"), (common + "train_8_db.tfrecords"), (common + "train_9_db.tfrecords")]
# Transforms a scalar string `example_proto` into a pair of a scalar string and
# a scalar integer, representing an image and its label, respectively.
def _parse_function(example_proto):
features = {
'height': tf.FixedLenFeature([], tf.int64),
'width': tf.FixedLenFeature([], tf.int64),
'image_raw': tf.FixedLenFeature([], tf.string),
'features': tf.FixedLenFeature([432], tf.float32)
}
parsed_features = tf.parse_single_example(example_proto, features)
# This is how we create one example, that is, extract one example from the database.
image = tf.decode_raw(parsed_features['image_raw'], tf.uint8)
# The height and the weights are used to
height = tf.cast(parsed_features['height'], tf.int32)
width = tf.cast(parsed_features['width'], tf.int32)
# The image is reshaped since when stored as a binary format, it is flattened. Therefore, we need the
# height and the weight to restore the original image back.
image = tf.reshape(image, [height, width, 3])
features = parsed_features['features']
return features, image
# Here is the temp var that I will use whenever the return batch from the dataset doesn't have a size of batch_size * 9 mentioned above.
random_features = tf.Variable(tf.zeros([72, 432], tf.float32))
random_images = tf.Variable(tf.zeros([72, 112, 112, 3], tf.uint8))
datasets = []
for _ in filenames:
datasets.append(tf.contrib.data.TFRecordDataset(_).map(_parse_function))
dataset_ziped = tf.contrib.data.TFRecordDataset.zip((datasets[0], datasets[1], datasets[2], datasets[3],
datasets[4], datasets[5], datasets[6], datasets[7], datasets[8]))
dataset = dataset_ziped.batch(batch_size)
iterator = dataset.make_initializable_iterator()
next_batch = iterator.get_next() # This has shape: [9, 2]
features = tf.concat((next_batch[0][0], next_batch[1][0], next_batch[2][0], next_batch[3][0],
next_batch[4][0], next_batch[5][0], next_batch[6][0], next_batch[7][0],
next_batch[8][0]), axis=0)
images = tf.concat((next_batch[0][1], next_batch[1][1], next_batch[2][1], next_batch[3][1],
next_batch[4][1], next_batch[5][1], next_batch[6][1], next_batch[7][1],
next_batch[8][1]), axis=0)
def get_features(features, images):
with tf.control_dependencies([tf.assign(random_features, features), tf.assign(random_images, images)]):
features = tf.reshape(features, shape=[9, 8, 432]) # where 8 * 9 = 72
features = tf.transpose(features, perm=[1, 0, 2]) # shape becomes: [8, 9, 432]
features = tf.reshape(features, shape=[72, 432]) # Now frames will be: 1st frame from 1st video, second from second video...
images = tf.reshape(images, shape=[9, 8, 112, 112, 3])
images = tf.transpose(images, perm=[1, 0, 2, 3, 4])
images = tf.reshape(images, shape=[72, 112, 112, 3])
return features, images
condition1 = tf.equal(tf.shape(features)[0], batch_size * 9)
condition2 = tf.equal(tf.shape(images)[0], batch_size * 9)
condition = tf.logical_and(condition1, condition2)
features, images = tf.cond(condition,
lambda: get_features(features, images),
lambda: get_features(random_features, random_images))
init_op = tf.global_variables_initializer()
with tf.Session() as sess:
# Initialize `iterator` with training data.
sess.run(init_op)
for _ in range(num_epoch):
sess.run(iterator.initializer)
# This while loop will run indefinitly until the end of the first epoch
while True:
try:
lst = []
features_np, images_np = sess.run([features, images])
for f in features_np:
lst.append(f[0])
print(lst)
except tf.errors.OutOfRangeError:
print('errorrrrr')
break
请注意，我总是提到batch_size * 9，因为在压缩数据集时，会创建一个样本数据，其中的元素是从9个不同的数据集中提取的。因此，由于我将batch_size指定为8，有72个样本，我拿了8 * 9。