Question

在我为每个卷积层添加xavier初始化后，损失开始变为负数。有人可以给出任何建议/理由吗？我将以下行添加到所有卷积层：

weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
          value: 0.1
        }

I0305 14:31:53.356343 11179 solver.cpp:219] Iteration 0 (-4.02766e+28 iter/s, 0.528933s/100 iters), loss = 2.05371
I0305 14:31:53.356374 11179 solver.cpp:238]     Train net output #0: accuracy = 0.11937
I0305 14:31:53.356384 11179 solver.cpp:238]     Train net output #1: loss = 2.05371 (* 1 = 2.05371 loss)
I0305 14:31:53.356395 11179 sgd_solver.cpp:105] Iteration 0, lr = 0.0001
I0305 14:32:28.728870 11179 solver.cpp:219] Iteration 100 (2.82699 iter/s, 35.3733s/100 iters), loss = 0.0270034
I0305 14:32:28.729014 11179 solver.cpp:238]     Train net output #0: accuracy = 1
I0305 14:32:28.729028 11179 solver.cpp:238]     Train net output #1: loss = 0 (* 1 = 0 loss)
I0305 14:32:28.729034 11179 sgd_solver.cpp:105] Iteration 100, lr = 0.0001
I0305 14:33:03.729997 11179 solver.cpp:219] Iteration 200 (2.85701 iter/s, 35.0017s/100 iters), loss = -8.27284e-09
I0305 14:33:03.730154 11179 solver.cpp:238]     Train net output #0: accuracy = 1
I0305 14:33:03.730167 11179 solver.cpp:238]     Train net output #1: loss = 0 (* 1 = 0 loss)
I0305 14:33:03.730172 11179 sgd_solver.cpp:105] Iteration 200, lr = 0.0001
I0305 14:33:38.885211 11179 solver.cpp:219] Iteration 300 (2.84449 iter/s, 35.1557s/100 iters), loss = -8.27284e-09
I0305 14:33:38.885368 11179 solver.cpp:238]     Train net output #0: accuracy = 1
I0305 14:33:38.885383 11179 solver.cpp:238]     Train net output #1: loss = 0 (* 1 = 0 loss)
I0305 14:33:38.885387 11179 sgd_solver.cpp:105] Iteration 300, lr = 0.0001
I0305 14:34:14.174548 11179 solver.cpp:219] Iteration 400 (2.83368 iter/s, 35.2898s/100 iters), loss = -8.27284e-09
I0305 14:34:14.174702 11179 solver.cpp:238]     Train net output #0: accuracy = 1
I0305 14:34:14.174720 11179 solver.cpp:238]     Train net output #1: loss = 0 (* 1 = 0 loss)
I0305 14:34:14.174724 11179 sgd_solver.cpp:105] Iteration 400, lr = 0.0001
I0305 14:34:49.578112 11179 solver.cpp:219] Iteration 500 (2.82453 iter/s, 35.4041s/100 iters), loss = -8.27284e-09
I0305 14:34:49.578254 11179 solver.cpp:238]     Train net output #0: accuracy = 1
I0305 14:34:49.578269 11179 solver.cpp:238]     Train net output #1: loss = 0 (* 1 = 0 loss)
I0305 14:34:49.578272 11179 sgd_solver.cpp:105] Iteration 500, lr = 0.0001
I0305 14:35:25.042238 11179 solver.cpp:219] Iteration 600 (2.81971 iter/s, 35.4646s/100 iters), loss = -8.27284e-09
I0305 14:35:25.042421 11179 solver.cpp:238]     Train net output #0: accuracy = 1
I0305 14:35:25.042438 11179 solver.cpp:238]     Train net output #1: loss = 0 (* 1 = 0 loss)
I0305 14:35:25.042443 11179 sgd_solver.cpp:105] Iteration 600, lr = 0.0001
I0305 14:36:00.540053 11179 solver.cpp:219] Iteration 700 (2.81704 iter/s, 35.4983s/100 iters), loss = -8.27284e-09
I0305 14:36:00.540194 11179 solver.cpp:238]     Train net output #0: accuracy = 1
I0305 14:36:00.540207 11179 solver.cpp:238]     Train net output #1: loss =

我的另一个问题是，在某些网络中，添加了Gaussian。像：

weight_filler {
   type: "gaussian"
   std: 0.005
}
bias_filler {
    type: "constant"
    value: 0.1
}

为什么我们将这些参数添加到卷积层？是吗因为我们正在从头开始训练网络？
如何将特定值分配给std和/或bias_filler 值？

我非常感谢你的帮助。

Answer 1

您的损失是-8.27284e-09，实际上是零并且不是负数（caffe使用单精度浮点数而不是双精度）。
你使用什么损耗层？ "SoftmaxWithLoss"

随机

bias_filler和wieght_filler个参数。如果您从现有模型开始训练（即微调），则这些参数没有任何意义。
std值是根据扇入和扇出（即通道内和通道数）计算的，以保持Blob值的统计值大致为零均值和单位差异。
您可以在Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (arXiv 2015)。

添加Xavier和bias_filler后，损失值开始变为负值。为什么？

1 个答案: