Question

在R中使用mlr包，我正在创建随机森林模型。为了评估模型的分类准确性，我使用了here中所述的嵌套重采样。我的问题是，内循环内的随机森林模型的分类精度通常比外循环结果高15％。我观察到内循环内的分类精度约为85％，但外循环的精度通常以70％左右结束。我无法在此处提供数据，但我正在粘贴正在使用的代码。

那怎么可能？可能是什么原因？

rf_param_set <- makeParamSet(
  ParamHelpers::makeDiscreteParam('mtry', values = c(3, 7, 14)),
  ParamHelpers::makeDiscreteParam('ntree', values = c(1000, 2000))
  )
rf_tune_ctrl <- makeTuneControlGrid()
rf_inner_resample <- makeResampleDesc('Bootstrap', iters = 5)
acc632plus <- setAggregation(acc, b632plus)
rf_learner <- makeTuneWrapper('classif.randomForest',
                              resampling = rf_inner_resample,
                              measures = list(acc),
                              par.set = rf_param_set,
                              control = rf_tune_ctrl,
                              show.info = TRUE)
# rf_outer_resample <- makeResampleDesc('Subsample', iters = 10, split = 2/3)
rf_outer_resample <- makeResampleDesc('Bootstrap', iters = 10, predict = 'both')
rf_result_resample <- resample(rf_learner, clf_task,
                               resampling = rf_outer_resample,
                               extract = getTuneResult,
                               measures = list(acc, acc632plus),
                               show.info = TRUE)

您可以在下面获得结果输出。

Resampling: OOB bootstrapping
Measures:             acc.train   acc.test    acc.test    
[Tune] Started tuning learner classif.randomForest for parameter set:
          Type len Def    Constr Req Tunable Trafo
mtry  discrete   -   -    3,7,14   -    TRUE     -
ntree discrete   -   - 1000,2000   -    TRUE     -
With control class: TuneControlGrid
Imputation value: -0
[Tune-x] 1: mtry=3; ntree=1000
[Tune-y] 1: acc.test.mean=0.8415307; time: 0.1 min
[Tune-x] 2: mtry=7; ntree=1000
[Tune-y] 2: acc.test.mean=0.8405726; time: 0.1 min
[Tune-x] 3: mtry=14; ntree=1000
[Tune-y] 3: acc.test.mean=0.8330845; time: 0.1 min
[Tune-x] 4: mtry=3; ntree=2000
[Tune-y] 4: acc.test.mean=0.8415809; time: 0.3 min
[Tune-x] 5: mtry=7; ntree=2000
[Tune-y] 5: acc.test.mean=0.8395083; time: 0.3 min
[Tune-x] 6: mtry=14; ntree=2000
[Tune-y] 6: acc.test.mean=0.8373584; time: 0.3 min
[Tune] Result: mtry=3; ntree=2000 : acc.test.mean=0.8415809
[Resample] iter 1:    0.9961089   0.7434555   0.7434555   
[Tune] Started tuning learner classif.randomForest for parameter set:
          Type len Def    Constr Req Tunable Trafo
mtry  discrete   -   -    3,7,14   -    TRUE     -
ntree discrete   -   - 1000,2000   -    TRUE     -
With control class: TuneControlGrid
Imputation value: -0
[Tune-x] 1: mtry=3; ntree=1000
[Tune-y] 1: acc.test.mean=0.8479891; time: 0.1 min
[Tune-x] 2: mtry=7; ntree=1000
[Tune-y] 2: acc.test.mean=0.8578465; time: 0.1 min
[Tune-x] 3: mtry=14; ntree=1000
[Tune-y] 3: acc.test.mean=0.8556608; time: 0.1 min
[Tune-x] 4: mtry=3; ntree=2000
[Tune-y] 4: acc.test.mean=0.8502869; time: 0.3 min
[Tune-x] 5: mtry=7; ntree=2000
[Tune-y] 5: acc.test.mean=0.8601446; time: 0.3 min
[Tune-x] 6: mtry=14; ntree=2000
[Tune-y] 6: acc.test.mean=0.8586638; time: 0.3 min
[Tune] Result: mtry=7; ntree=2000 : acc.test.mean=0.8601446
[Resample] iter 2:    0.9980545   0.7032967   0.7032967   
[Tune] Started tuning learner classif.randomForest for parameter set:
          Type len Def    Constr Req Tunable Trafo
mtry  discrete   -   -    3,7,14   -    TRUE     -
ntree discrete   -   - 1000,2000   -    TRUE     -
With control class: TuneControlGrid
Imputation value: -0
[Tune-x] 1: mtry=3; ntree=1000
[Tune-y] 1: acc.test.mean=0.8772566; time: 0.1 min
[Tune-x] 2: mtry=7; ntree=1000
[Tune-y] 2: acc.test.mean=0.8750990; time: 0.1 min
[Tune-x] 3: mtry=14; ntree=1000
[Tune-y] 3: acc.test.mean=0.8730733; time: 0.1 min
[Tune-x] 4: mtry=3; ntree=2000
[Tune-y] 4: acc.test.mean=0.8782829; time: 0.3 min
[Tune-x] 5: mtry=7; ntree=2000
[Tune-y] 5: acc.test.mean=0.8741619; time: 0.3 min
[Tune-x] 6: mtry=14; ntree=2000
[Tune-y] 6: acc.test.mean=0.8687918; time: 0.3 min
[Tune] Result: mtry=3; ntree=2000 : acc.test.mean=0.8782829
[Resample] iter 3:    0.9902724   0.7329843   0.7329843

Answer 1

您所看到的正是您要使用嵌套重采样的原因-内部重采样循环对数据过度拟合（在某种程度上），并给人以误导性的印象，泛化了性能。在进行外部重采样后，您可以检测到这一点（准确性较低）。

mlr教程对此页面（https://mlr.mlr-org.com/articles/tutorial/nested_resampling.html）有更详细的介绍。通常，您看不到这些结果是因为您做错了任何事情（除非您以某种方式手动拆分了数据），而只是使用了一种强大的优化方法，该方法比应进行的优化更多—您正在通过嵌套重采样检测到这一点。

您可以尝试使用交叉验证而非引导程序；这可能会提供更一致的结果。

在嵌套重采样中，分类准确性结果发生了巨大变化

1 个答案: