im2txt

Question

我在为im2txt微调预先训练的Inception v3模型时遇到问题。出于某种原因，初始训练并没有大大减少损失，微调初始v3并没有减少我的训练数据的任何损失。我试图找出原因，任何见解都会有所帮助。

im2txt是一个模型，它接收图像输入并打印出一系列字幕作为输出。最初，im2txt将标题打印为连贯句子，描述图像。为了适应我的项目，我更改了训练数据中的代码和标签，以便打印出与图像相关的单词列表。

例如，我的图片看起来像这样。请注意，图像中有很多对象而不是普通的Imagenet图像：

我的标签字幕如下所示：

 female lady woman clothes shop customer

我总共有400,000张图片和相应的标签标题。我训练了130,000步的初步训练，并进行了170,000步的微调。词汇总共有750个单词。初始训练+微调（从步骤130,000开始）的损失曲线如下：

精度和召回率约为0.35~40。

培训的配置文件如下：

# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

"""Image-to-text model and training configurations."""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function


class ModelConfig(object):
  """Wrapper class for model hyperparameters."""

  def __init__(self):
    """Sets the default model hyperparameters."""
    # File pattern of sharded TFRecord file containing SequenceExample protos.
    # Must be provided in training and evaluation modes.
    self.input_file_pattern = None

    # Image format ("jpeg" or "png").
    self.image_format = "jpeg"

    # Approximate number of values per input shard. Used to ensure sufficient
    # mixing between shards in training.
    self.values_per_input_shard = 2300
    # Minimum number of shards to keep in the input queue.
    self.input_queue_capacity_factor = 2
    # Number of threads for prefetching SequenceExample protos.
    self.num_input_reader_threads = 1

    # Name of the SequenceExample context feature containing image data.
    self.image_feature_name = "image/data"
    # Name of the SequenceExample feature list containing integer captions.
    self.caption_feature_name = "image/caption_ids"

    # Number of unique words in the vocab (plus 1, for <UNK>).
    # The default value is larger than the expected actual vocab size to allow
    # for differences between tokenizer versions used in preprocessing. There is
    # no harm in using a value greater than the actual vocab size, but using a
    # value less than the actual vocab size will result in an error.
    self.vocab_size = 750

    # Number of threads for image preprocessing. Should be a multiple of 2.
    self.num_preprocess_threads = 4

    # Batch size.
    self.batch_size = 32

    # File containing an Inception v3 checkpoint to initialize the variables
    # of the Inception model. Must be provided when starting training for the
    # first time.
    self.inception_checkpoint_file = None

    # Dimensions of Inception v3 input images.
    self.image_height = 299
    self.image_width = 299

    # Scale used to initialize model variables.
    self.initializer_scale = 0.08

    # LSTM input and output dimensionality, respectively.
    self.embedding_size = 512
    self.num_lstm_units = 512

    # If < 1.0, the dropout keep probability applied to LSTM variables.
    self.lstm_dropout_keep_prob = 0.7


class TrainingConfig(object):
  """Wrapper class for training hyperparameters."""

  def __init__(self):
    """Sets the default training hyperparameters."""
    # Number of examples per epoch of training data.
    self.num_examples_per_epoch = 100000

    # Optimizer for training the model.
    self.optimizer = "SGD"

    # Learning rate for the initial phase of training.
    self.initial_learning_rate = 2.0
    self.learning_rate_decay_factor = 0.5
    self.num_epochs_per_decay = 1.0

    # Learning rate when fine tuning the Inception v3 parameters.
    self.train_inception_learning_rate = 0.005

    # If not None, clip gradients to this value.
    self.clip_gradients = 5.0

    # How many model checkpoints to keep.
    self.max_checkpoints_to_keep = 5

任何建议，见解或观察都会很棒。

Answer 1

请注意，im2txt在创建可读句子的意义上非常强大。在句子中，一个单词与相邻单词相关，这就是它起作用的原因。在您的情况下，您正在更改模型以生成一组与订单无关的标签。实际上，在im2txt模型中，“女性”，“女性”和“女性”这两个词的概念基本相同，im2txt可以创建滑动这个词的句子。例如，在im2txt中：“那位女士穿着漂亮的粉红色裙子”就像“女人的裙子是粉红色的”一样，或者应该非常相似。在您的情况下，如果您没有为单词的顺序提供一些规则，那么它会使您的模型混淆很多，并且可能无法学习。

如果要从图像中获取标签列表，则应仅使用具有多标记分类的初始模型（通过sigmoid层更改softmax图层并使用sigmoid cross entropy作为损失函数）。