Tensorflow 2抛出ValueError:未知TensorShape上未定义as_list()

时间:2019-10-29 11:19:39

标签: python tensorflow image-segmentation tensorflow-datasets

我正在尝试在Tensorflow 2.0中训练一个Unet模型,该模型将图像和分割蒙版作为输入,但是我得到了ValueError : as_list() is not defined on an unknown TensorShape。堆栈跟踪显示问题在_get_input_from_iterator(inputs)期间发生:

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in _prepare_feed_values(model, inputs, mode)
    110     for inputs will always be wrapped in lists.
    111   """
--> 112   inputs, targets, sample_weights = _get_input_from_iterator(inputs)
    113 
    114   # When the inputs are dict, then we want to flatten it in the same order as

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in _get_input_from_iterator(iterator)
    147   # Validate that all the elements in x and y are of the same type and shape.
    148   dist_utils.validate_distributed_dataset_inputs(
--> 149       distribution_strategy_context.get_strategy(), x, y, sample_weights)
    150   return x, y, sample_weights
    151 

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py in validate_distributed_dataset_inputs(distribution_strategy, x, y, sample_weights)
    309 
    310   if y is not None:
--> 311     y_values_list = validate_per_replica_inputs(distribution_strategy, y)
    312   else:
    313     y_values_list = None

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py in validate_per_replica_inputs(distribution_strategy, x)
    354     if not context.executing_eagerly():
    355       # Validate that the shape and dtype of all the elements in x are the same.
--> 356       validate_all_tensor_shapes(x, x_values)
    357     validate_all_tensor_types(x, x_values)
    358 

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py in validate_all_tensor_shapes(x, x_values)
    371 def validate_all_tensor_shapes(x, x_values):
    372   # Validate that the shape of all the elements in x have the same shape
--> 373   x_shape = x_values[0].shape.as_list()
    374   for i in range(1, len(x_values)):
    375     if x_shape != x_values[i].shape.as_list():

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/tensor_shape.py in as_list(self)
   1169     """
   1170     if self._dims is None:
-> 1171       raise ValueError("as_list() is not defined on an unknown TensorShape.")
   1172     return [dim.value for dim in self._dims]
   1173

我已经浏览了其他两个Stackoverflow帖子(herehere),但都出现了此错误,但就我而言,我认为问题出在传递给DataSet的map函数中。我将下面定义的process_path函数称为tensorflow DataSet的map函数。这接受到图像的路径,并构建到相应的numpy file分段掩码的路径。然后,使用kerasUtil.to_categorical将numpy文件中的(256 256)数组转换为(256 256 10),其中10个通道代表每个类。我使用check_shape函数来确认张量形状正确,但是当我调用model.fit时仍无法得出该形状。

# --------------------------------------------------------------------------------------
# DECODE A NUMPY .NPY FILE INTO THE REQUIRED FORMAT FOR TRAINING
# --------------------------------------------------------------------------------------
def decode_npy(npy):
  filename = npy.numpy()
  data = np.load(filename)
  data = kerasUtils.to_categorical(data, 10)
  return data

def check_shape(image, mask):
  print('shape of image: ', image.get_shape())
  print('shape of mask: ', mask.get_shape())
  return 0.0

# --------------------------------------------------------------------------------------
# DECODE AN IMAGE (PNG) FILE INTO THE REQUIRED FORMAT FOR TRAINING
# --------------------------------------------------------------------------------------
def decode_img(img):
  # convert the compressed string to a 3D uint8 tensor
  img = tf.image.decode_png(img, channels=3)
  # Use `convert_image_dtype` to convert to floats in the [0,1] range.
  return tf.image.convert_image_dtype(img, tf.float32)

# --------------------------------------------------------------------------------------
# PROCESS A FILE PATH FOR THE DATASET
# input - path to an image file
# output - an input image and output mask
# --------------------------------------------------------------------------------------
def process_path(filePath):
  parts = tf.strings.split(filePath, '/')
  fileName = parts[-1]
  parts = tf.strings.split(fileName, '.')
  prefix = tf.convert_to_tensor(convertedMaskDir, dtype=tf.string)
  suffix = tf.convert_to_tensor("-mask.npy", dtype=tf.string)
  maskFileName = tf.strings.join((parts[-2], suffix))
  maskPath = tf.strings.join((prefix, maskFileName), separator='/')

  # load the raw data from the file as a string
  img = tf.io.read_file(filePath)
  img = decode_img(img)
  mask = tf.py_function(decode_npy, [maskPath], tf.float32)

  return img, mask

# --------------------------------------------------------------------------------------
# CREATE A TRAINING and VALIDATION DATASETS
# --------------------------------------------------------------------------------------
trainSize = int(0.7 * DATASET_SIZE)
validSize = int(0.3 * DATASET_SIZE)

allDataSet = tf.data.Dataset.list_files(str(imageDir + "/*"))
# allDataSet = allDataSet.map(process_path, num_parallel_calls=AUTOTUNE)
# allDataSet = allDataSet.map(process_path)

trainDataSet = allDataSet.take(trainSize)
trainDataSet = trainDataSet.map(process_path).batch(64)
validDataSet = allDataSet.skip(trainSize)
validDataSet = validDataSet.map(process_path).batch(64)

...

# this code throws the error!
model_history = model.fit(trainDataSet, epochs=EPOCHS,
                          steps_per_epoch=stepsPerEpoch,
                          validation_steps=validationSteps,
                          validation_data=validDataSet,
                          callbacks=callbacks)

3 个答案:

答案 0 :(得分:1)

我在处理图像和遮罩时遇到了与您相同的问题,并通过在预处理功能期间手动设置它们的形状来解决了问题,特别是在tf.map期间调用pyfunc时。

def process_path(filePath):
  ...

  # load the raw data from the file as a string
  img = tf.io.read_file(filePath)
  img = decode_img(img)
  mask = tf.py_function(decode_npy, [maskPath], tf.float32)

  # TODO:
  img.set_shape([MANUALLY ENTER THIS])
  mask.set_shape([MANUALLY ENTER THIS])

  return img, mask

答案 1 :(得分:1)

我遇到了类似的问题,并使用dataset.padded_batch和显式padded_shapes解决了我的问题!

答案 2 :(得分:0)

我遇到了同样的问题,在我的管道中还有一个 py_func 映射。

事实证明,该问题与损失函数和指标之间的兼容性有关。我试图将“分类交叉熵”损失与“准确性”指标结合使用。

通过将指标更改为“分类准确率”解决了我的问题。