Question

这是一个简单的问题，只是为了确保我不会这么做。我希望使用auc作为我在mlr中的度量，并且由于样本量较小，我也使用LOO。当然，在LOO交叉验证方案中，测试样本始终只有一个实例，因此无法计算auc。当然，我们可以在之后计算它，查看预测，当我们想要将它用作嵌套交叉验证的内循环中的度量时，问题就会发生。像这样的东西（你必须定义自己的binaryTask）：

require(mlr)    
#for example purposes we will decide which one is better, vanilla LDA or
#vanilla SVM, in the task specified below
bls = list(makeLearner("classif.lda"),makeLearner("classif.svm"))
#modelMultiplexer allows us to search whole parameter spaces between models
#as if the models themselves were parameters
lrn = makeModelMultiplexer(bls)
#to calculate AUC we need some continuous output, so we set 
#predictType to probabilities
lrn = setPredictType(lrn, "prob")
lrn = setId(lrn, "Model Multiplexer")
#here we could pass the parameters to be tested to both SVM and LDA,
#let's not pass anything so we test the vanilla classifiers instead
ps = makeModelMultiplexerParamSet(lrn)
#finally, the resample strategy, Leave-One-Out ("LOO") in our case
rdesc = makeResampleDesc("LOO")
#parameter space search strategy, in our case we only have one parameter:
#the model. So, a simple grid search will do the trick
ctrl = makeTuneControlGrid()
#The inner CV loop where we choose the best model in the validation data
tune = makeTuneWrapper(lrn, rdesc, par.set = ps, control = ctrl, measure = auc, show.info = FALSE) 
#The outer CV loop where we obtain the performace of the selected model 
#in the test data. mlR is a great interface, we could have passed a list 
#of classifiers and tasks here instead and do it all in one go 
#(beware your memory limitation!)
res = benchmark(tune, binaryTask, rdesc, measure = auc)

在这两个循环中，您根本无法使用auc。我们怎样才能mlr每次评估所有测试样本的度量而不是唯一的重新采样？

Answer 1

您可以对内循环使用不同的重采样策略，然后使用auc：

library(mlr)    

ps = makeParamSet(
  makeNumericLearnerParam(id = "cp", default = 0.01, lower = 0, upper = 1)
)
ctrl = makeTuneControlRandom(maxit = 10)
inner = makeResampleDesc("Subsample")
lrn = makeLearner("classif.rpart", predict.type = "prob")
tune = makeTuneWrapper(lrn, resampling = inner, par.set = ps, control = ctrl, measure = auc)

outer = makeResampleDesc("LOO")
r = resample(tune, bc.task, resampling = outer, extract = getTuneResult, measure = auc)

您还可以获取重新采样结果并在其上计算任意性能指标，例如： performance(r$pred, auc)。

计算mlR中的AUC Leave-One-Out交叉验证？

1 个答案: