Question

我有两个数据集，并使用caffe库进行CNN训练。

第一个数据集包含大量训练数据，超过60,000张训练图像和16,000张测试图像。它的求解器文件可以看如下。培训中的批量大小设置为32。

train_net: "/home/Softwares/Projects/caffe-ssd-2/NumberPlate/InceptionNet/6/train_0.prototxt"
test_net: "/home/Softwares/Projects/caffe-ssd-2/NumberPlate/InceptionNet/6/test_0.prototxt"
test_iter: 2080
test_interval: 4000
base_lr: 0.0010000000475
display: 10
max_iter: 16000
lr_policy: "multistep"
gamma: 0.10000000149
momentum: 0.899999976158
weight_decay: 0.000500000023749
snapshot: 2000
snapshot_prefix: "/home/Softwares/Projects/caffe-ssd-2/NumberPlate/InceptionNet/6/InceptionNet"
solver_mode: GPU
device_id: 0
debug_info: false
snapshot_after_train: true
test_initialization: false
average_loss: 10
stepvalue: 4000
stepvalue: 8000
stepvalue: 12000
iter_size: 1
momentum2: 0.999000012875
type: "Adam"
eval_type: "detection"
ap_version: "11point"
num_total_train_images: 62308
pathtolog: "/home/Softwares/Projects/caffe-ssd-2/NumberPlate/InceptionNet/6"
batchsize: 32
meanprecision: 0.5
scratch: 1

我有第二组数据，火车图像数量较少。如下将仅设置2883个训练图像和709个测试图像以及用于训练的批次大小。

train_net: "/home /Softwares/Projects/caffe-ssd-2/Nextan/InceptionNet/0/train_0.prototxt"
test_net: "/home/Softwares/Projects/caffe-ssd-2/Nextan/InceptionNet/0/test_0.prototxt"
test_iter: 177
test_interval: 500
base_lr: 0.0010000000475
display: 10
max_iter: 8000
lr_policy: "multistep"
gamma: 0.10000000149
momentum: 0.899999976158
weight_decay: 0.000500000023749
snapshot: 1000
snapshot_prefix: "/home/Softwares/Projects/caffe-ssd-2/Nextan/InceptionNet/0/InceptionNet"
solver_mode: GPU
device_id: 0
debug_info: false
snapshot_after_train: true
test_initialization: false
average_loss: 10
stepvalue: 2000
stepvalue: 4000
stepvalue: 6000
iter_size: 1
momentum2: 0.999000012875
type: "Adam"
eval_type: "detection"
ap_version: "11point"
num_total_train_images: 2883
pathtolog: "/home/Softwares/Projects/caffe-ssd-2/Nextan/InceptionNet/0"
batchsize: 16
meanprecision: 0.5
scratch: 1

我在具有相同GPU和资源的同一台PC上进行了培训。第二个数据集给了我"Check failed: error == cudaSuccess (74 vs. 0) misaligned address" 但是第一个数据集已成功训练。怎么了？

Answer 1

这是Caffe中的内部错误，原因是在某些情况下max_workspace不是16的倍数，并且导致工作区在内存中未对齐。我要尝试的第一件事是更改批处理大小。

在这里您可以看到与该问题有关的请求请求： https://github.com/BVLC/caffe/pull/6548

检查失败：错误== cudaSuccess（74和0）地址未对齐

1 个答案: