Question

我正在尝试使用.config文件中的data_augmentation_options来训练网络，特别是ssd_mobilenet_v1，但是当我激活选项random_adjust_brightness时，我会很快得到下面粘贴的错误消息（我在步骤110000之后激活了该选项））。

我尝试减少默认值：

optional float max_delta=1 [default=0.2];

但结果是一样的。

知道为什么吗？图像是来自png文件的RGB（来自Bosch Small Traffic Lights Dataset）。

INFO:tensorflow:global step 110011: loss = 22.7990 (0.357 sec/step)
INFO:tensorflow:global step 110012: loss = 47.8811 (0.401 sec/step)
2017-11-16 11:02:29.114785: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.114895: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.114969: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.115043: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.115112: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
...

编辑：我找到的解决方法就是这个。 inf或nan是丢失的，所以检查/object_detection/core/preprocessor.py中的函数进行亮度随机化：

def random_adjust_brightness(image, max_delta=0.2):
  """Randomly adjusts brightness.

  Makes sure the output image is still between 0 and 1.

  Args:
    image: rank 3 float32 tensor contains 1 image -> [height, width, channels]
           with pixel values varying between [0, 1].
    max_delta: how much to change the brightness. A value between [0, 1).

  Returns:
    image: image which is the same shape as input image.
    boxes: boxes which is the same shape as input boxes.
  """
  with tf.name_scope('RandomAdjustBrightness', values=[image]):
    image = tf.image.random_brightness(image, max_delta)
    image = tf.clip_by_value(image, clip_value_min=0.0, clip_value_max=1.0)
    return image

假设图像值必须介于0.0和1.0之间。图像是否可能实际到达0均值甚至不同范围？在这种情况下，剪辑会破坏它们并导致失败。长话短说：我评论了剪切线，它正在工作（我们将看到结果）。

Answer 1

通常，获取LossTensor is inf or nan. : Tensor had NaN values是由于边界框/注释中的错误（来源：https://github.com/tensorflow/models/issues/1881）。

我知道博世小交通灯数据集有一些注释超出了图像尺寸。例如，该数据集中图像的高度为720像素，但是某些边界框的高度坐标大于720.这很常见，因为无论何时记录序列的汽车都在交通信号灯下，一些交通信号灯都可见，其中一些被切断了。

我知道这不是您问题的准确答案，但希望能够提供有关您遇到问题的可能原因的见解。也许删除延伸到图像尺寸之外的注释将有助于解决问题;但是，我正在处理同样的问题，除了我没有使用图像预处理。在同一个数据集上，我每隔8000步就遇到LossTensor is inf or nan. : Tensor had NaN values错误。

Answer 2

Addition to annotations that extend outside the image dimensions, Bosch Traffic Light detection training dataset also has one image where x_max < x_min and y_max < y_min which causes a negative width and height. This causes the "LossTensor is inf or nan. : Tensor had NaN values" error every ~8000 steps. I had the same error; removing the problematic entries resolved the issue.

Answer 3

我也碰到了这个，我最后写了一个快速而又脏的脚本来找到坏蛋。我不知道图像集是否随时间而变化，但我下载的集合有三个带注释的图像 ./rgb/train/2015-10-05-11-26-32_bag/105870.png

./ RGB /火车/ 2015-10-05-11-26-32_bag / 108372.png

./ RGB /火车/ 2015-10-05-14-40-46_bag / 462350.png

对于那些感兴趣的人，继承我的剧本：

import yaml
import os

INPUT_YAML = "train.yaml"
examples = yaml.load(open(INPUT_YAML, 'rb').read())
len_examples = len(examples)
print("Loaded ", len(examples), "examples")
for example in examples:
  for box in example['boxes']:
    xmin = float(box['x_min'])
    xmax = float(box['x_max'])
    ymin = float(box['y_min'])
    ymax = float(box['y_max'])
    if xmax < xmin or xmax > 1280 or xmin > 1280:
      print( "INVALID IMAGE: ", example['path'], " X_MAX = ", float(box['x_max']) )
    if ymax < ymin or ymax > 720 or ymin > 720:
      print( "INVALID IMAGE: ", example['path'], " Y_MAX = ", float(box['y_max']) )

在Object Detection API中使用数据扩充选项时出错

3 个答案: