Question

尝试运行基于Tensorflow对象检测API的程序。更快的RCNN模型停止了GCP训练，但在本地运行没有问题。对于任何反馈，我们都表示感谢。尝试了不同帖子中建议的Service Agent的Logs Writer角色权限。找不到更多反馈。

完整错误消息：

副本母版0以非零状态1退出。原因：错误。追溯（最近一次通话）：文件 _run_module_as_main中的“ /usr/lib/python2.7/runpy.py”，第174行 “ 主要”，fname，加载程序，pkg_name）文件 _run_code执行代码中的“ /usr/lib/python2.7/runpy.py”，第72行 run_globals文件 “ /root/.local/lib/python2.7/site-packages/object_detection/train.py”， tf.app.run（）文件中的第198行 “ /usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py”，第48行，在运行中_sys.exit（main（_sys.argv [：1] + flags_passthrough））文件 “ /root/.local/lib/python2.7/site-packages/object_detection/train.py”，第194行，位于主worker_job_name，is_chief，FLAGS.train_dir中） “ /root/.local/lib/python2.7/site-packages/object_detection/trainer.py”，第296行，在火车saver = saver中） “ /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py”，火车sess，train_op，global_step，train_step_kwargs中的763行）文件 “ /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py”，在train_step run_metadata = run_metadata中的第487行）文件 “ /usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py”，第889行，在运行run_metadata_ptr）文件中 “ /usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py”，第1120行，在_run feed_dict_tensor中，选项，run_metadata）文件 “ /usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py”，第1317行，在_do_run选项中，run_metadata）文件 “ /usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py”，第1336行，在_do_call中，引发类型（e）（node_def，op，message） UnavailableError：端点读取失败，以了解有关您的原因的更多信息作业已退出，请检查日志： https://console.cloud.google.com/logs/viewer?project=1086278442266&resource=ml_job%2Fjob_id%2Fuav_object_detection_1543356760&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22uav_object_detection_1543356760%22

这是我在终端机中开始训练的内容：

gcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` \
   --job-dir=gs://my_gcs_bucket/train \
   --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz \
   --module-name object_detection.train \
   --region us-central1 \
   --config object_detection/samples/cloud/cloud.yml \
   --runtime-version=1.4 \
   -- \
   --train_dir=gs://my_gcs_bucket/train \
   --pipeline_config_path=gs://my_gcs_bucket/data/faster_rcnn_resnet101.config

这是我在GCP存储桶中的文件结构

+ data/
  - faster_rcnn_resnet101.config
  - model.ckpt.index
  - model.ckpt.meta
  - model.ckpt.data-00000-of-00001
  - pet_label_map.pbtxt
  - train.record
  - val.record
+ train/

这是我正在运行的文件夹中的文件结构

+dist/
  -object_detection-0.1.tar.gz
+object_detection/
+object_detection.egg-info/
+slim/
setup.py

配置文件：

# Faster R-CNN with Resnet-101 (v1) configured for the Oxford-IIIT Pet Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.

model {
  faster_rcnn {
    num_classes: 1
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 600
        max_dimension: 1024
      }
    }
    feature_extractor {
      type: 'faster_rcnn_resnet101'
      first_stage_features_stride: 16
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 16
        width_stride: 16
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 300
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}

train_config: {
  batch_size: 1
  batch_queue_capacity: 1
  num_batch_queue_threads: 1
  prefetch_queue_capacity: 1
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 0
            learning_rate: .0003
          }
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "gs://my_gcs_bucket/data/model.ckpt"
  from_detection_checkpoint: true
  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the pets dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.
  num_steps:2000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: "gs://my_gcs_bucket/data/data/train.record"
  }
  label_map_path: "gs://my_gcs_bucket/data/data/label_map.pbtxt"
  queue_capacity: 10
  min_after_dequeue: 5
}

eval_config: {
  num_examples: 4
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "gs://my_gcs_bucket/data/data/val.record"
  }
  label_map_path: "gs://my_gcs_bucket/data/data/label_map.pbtxt"
  shuffle: false
  num_readers: 1
}

Answer 1

在cloud.yml和初始请求中将运行时版本更改为1.2。

更快的RCNN模型训练停止在GCP上运行，在本地运行而没有问题

1 个答案: