Question

我正在尝试针对动力学数据集的子集训练一个简单的3DCNN以进行动作分类。我正在传递tf.data.Dataset.from_generator（）对象作为对model.fit（）的调用中的输入。

tensorflow版本：r1.12

初始化tf.data.Dataset的生成器产生np.arrays元组。第一个是形状为（50,45,80,3）的预处理视频，第二个是形状为（22，）的类的一次性编码

代码：

import os
import numpy as np
import itertools

import tensorflow as tf
import tensorflow.data as data
from tensorflow.keras.models import Sequential 
from tensorflow.keras.layers import MaxPooling3D, Conv3D, BatchNormalization, Dense 
from tensorflow.keras.layers import Dropout, Activation, Flatten, Input


def train_generator():
    train_dir = '/home/kjd/Storage/kinetics-frames_proc_small'
    classes = os.listdir(train_dir)
    for index, label in enumerate(classes):
        clips = os.listdir(train_dir + '/' + label)
        for clip in clips:
            data = np.load(train_dir + '/' + label + '/' + clip)
            yield data, np.eye(22)[index].astype(int)


EPOCHS = 3
BATCH_SIZE = 32
dataset = data.Dataset.from_generator(train_generator, (tf.int64, tf.int64))



model = Sequential()
model.add(Conv3D(16, (3,3,3), strides=(1,1,1), padding='same', activation='relu',
                 input_shape=(50,45,80,3)))
model.add(Conv3D(32, (3,3,3), strides=(1,1,1), padding='same', activation='relu'))
model.add(MaxPooling3D(pool_size=(2,2,2), strides=(2,2,2)))
model.add(BatchNormalization())
model.add(Conv3D(64, (3,3,3), strides=(1,1,1), padding='same', activation='relu'))
model.add(Conv3D(128, (3,3,3), strides=(1,1,1), padding='same', activation='relu'))
model.add(MaxPooling3D(pool_size=(2,2,2), strides=(2,2,2)))
model.add(BatchNormalization())
model.add(Conv3D(256, (3,3,3), strides=(1,1,1), padding='same', activation='relu'))
model.add(Conv3D(512, (3,3,3), strides=(1,1,1), padding='same', activation='relu'))
model.add(MaxPooling3D(pool_size=(2,2,2), strides=(2,2,2)))
model.add(BatchNormalization())
model.add(Flatten())
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(22, activation='softmax'))


model.compile('adam', 'categorical_crossentropy', metrics=['accuracy'])
model.fit(dataset, batch_size=BATCH_SIZE, epochs=EPOCHS, shuffle=False,
          steps_per_epoch=1000)

错误：

Traceback (most recent call last):
  File "train.py", line 55, in <module>
    steps_per_epoch=1000)
  File "/home/kjd/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1683, in fit
    shuffle=shuffle)
  File "/home/kjd/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1200, in _standardize_user_data
    class_weight, batch_size)
  File "/home/kjd/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1328, in _standardize_weights
    exception_prefix='input')
  File "/home/kjd/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py", line 294, in standardize_input_data
    data = [standardize_single_array(x) for x in data]
  File "/home/kjd/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py", line 294, in <listcomp>
    data = [standardize_single_array(x) for x in data]
  File "/home/kjd/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py", line 228, in standardize_single_array
    if x.shape is not None and len(x.shape) == 1:
  File "/home/kjd/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py", line 745, in __len__
    raise ValueError("Cannot take the length of shape with unknown rank.")
ValueError: Cannot take the length of shape with unknown rank.

看来tf.keras不喜欢我输入数据的格式。我对tf / keras还是相当陌生，但是从此错误消息中并未收获很多。如果有人对问题出在哪里有任何见识，您的想法将不胜感激。

Answer 1

我只是在尝试分发我的

时遇到了类似的问题

<DatasetV1Adapter shapes: <unknown>, types: tf.float32>" dataset using strategy.experimental_distribute_dataset() with tf.distribute.MirroredStrategy() as strategy. I got the same error as above (" raise ValueError("Cannot take the length of shape with unknown rank ValueError：无法采用未知等级的形状长度。 “）对于遇到类似问题的人，我的解决方案是使用DatasetV1Adapter数据集并使用data.Dataset.from_generator创建新数据集，如下所示：

        def generator(dataset):
            # dataset of type DatasetV1Adapter 
            for datapoint in dataset:
                yield datapoint

    dataset = tf.data.Dataset.from_generator(generator, (tf.float32), output_shapes=([None, None, None, None]))

dataset_dist = strategy.experimental_distribute_dataset(dataset)

为我工作！

Answer 2

我最近遇到这个问题；您可能需要提供output_shapes参数：

dataset = data.Dataset.from_generator(train_generator, (tf.int64, tf.int64), output_shapes=(tf.TensorShape([None, None, None, None]), tf.TensorShape([None])))

假设一个4维输入图像和一个1维输出数组。

tf.data.Dataset对象作为tf.Keras模型的输入-ValueError

2 个答案: