Question

在训练我的模型超过1个历元时出现以下错误

[02/06/2019 13:37:08 WARNING 140231582721856] Expected number of batches: 15, did not match the number of batches processed: 16. This may happen when some images or annotations are invalid and cannot be parsed. Please check the dataset and ensure it follows the format in the documentation.
[02/06/2019 13:37:08 INFO 140231582721856] #quality_metric: host=algo-1, epoch=24, batch=16 train cross_entropy <loss>=(nan)
[02/06/2019 13:37:08 INFO 140231582721856] #quality_metric: host=algo-1, epoch=24, batch=16 train smooth_l1 <loss>=(nan)
[02/06/2019 13:37:08 INFO 140231582721856] Round of batches complete
[02/06/2019 13:37:08 INFO 140231582721856] Updated the metrics
[02/06/2019 13:37:08 INFO 140231582721856] #quality_metric: host=algo-1, epoch=24, validation mAP <score>=(0.0)
[02/06/2019 13:37:08 INFO 140231582721856] #progress_metric: host=algo-1, completed 83 % of epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Batches Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Records Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Batches Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Records Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Max Records Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Reset Count": {"count": 1, "max": 25, "sum": 25.0, "min": 25}}, "EndTime": 1549460228.963195, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/Object Detection", "epoch": 24}, "StartTime": 1549460224.644808}

以下是我使用的代码

估算器

od_model = sagemaker.estimator.Estimator(training_image,
                                         role, 
                                         train_instance_count=1, 
                                         train_instance_type='ml.p3.8xlarge',
                                         train_volume_size = 500,
                                         train_max_run = 300000,
                                         input_mode= 'File',
                                         output_path=s3_output_location,
                                         sagemaker_session=sess)

对于超参数

od_model.set_hyperparameters(base_network='resnet-50',
                             use_pretrained_model=0,
                             num_classes=1,
                             mini_batch_size=32,
                             epochs=30,
                             learning_rate=0.001,
                             lr_scheduler_step='3,6',
                             lr_scheduler_factor=0.1,
                             optimizer='sgd',
                             momentum=0.9,
                             weight_decay=0.0005,
                             overlap_threshold=0.5,
                             nms_threshold=0.45,
                             image_shape=512,
                             label_width=360,
                             num_training_samples=500)

尽管输出模型无法正确检测并在各处创建框，但是如果我将epoch保持为1，则边界框会很好显示

使用上述代码，最终模型不会创建任何边界框

Answer 1

两个训练损失为“ nan”，且验证mAP为0。这意味着该模型未正确训练。尝试调整“ learning_rate”和“ batch_size”超参数。对于小型数据集（500张图像），您可以通过设置“ use_pretrained_model = 1”来使用转移学习功能。

Amazon Sagemaker对象检测的预期批处理错误数

1 个答案: