计算mlR中的AUC Leave-One-Out交叉验证?

时间:2015-10-20 20:16:38

标签: r classification cross-validation auc mlr

这是一个简单的问题,只是为了确保我不会这么做。我希望使用auc作为我在mlr中的度量,并且由于样本量较小,我也使用LOO。当然,在LOO交叉验证方案中,测试样本始终只有一个实例,因此无法计算auc。当然,我们可以在之后计算它,查看预测,当我们想要将它用作嵌套交叉验证的内循环中的度量时,问题就会发生。像这样的东西(你必须定义自己的binaryTask):

require(mlr)    
#for example purposes we will decide which one is better, vanilla LDA or
#vanilla SVM, in the task specified below
bls = list(makeLearner("classif.lda"),makeLearner("classif.svm"))
#modelMultiplexer allows us to search whole parameter spaces between models
#as if the models themselves were parameters
lrn = makeModelMultiplexer(bls)
#to calculate AUC we need some continuous output, so we set 
#predictType to probabilities
lrn = setPredictType(lrn, "prob")
lrn = setId(lrn, "Model Multiplexer")
#here we could pass the parameters to be tested to both SVM and LDA,
#let's not pass anything so we test the vanilla classifiers instead
ps = makeModelMultiplexerParamSet(lrn)
#finally, the resample strategy, Leave-One-Out ("LOO") in our case
rdesc = makeResampleDesc("LOO")
#parameter space search strategy, in our case we only have one parameter:
#the model. So, a simple grid search will do the trick
ctrl = makeTuneControlGrid()
#The inner CV loop where we choose the best model in the validation data
tune = makeTuneWrapper(lrn, rdesc, par.set = ps, control = ctrl, measure = auc, show.info = FALSE) 
#The outer CV loop where we obtain the performace of the selected model 
#in the test data. mlR is a great interface, we could have passed a list 
#of classifiers and tasks here instead and do it all in one go 
#(beware your memory limitation!)
res = benchmark(tune, binaryTask, rdesc, measure = auc)

在这两个循环中,您根本无法使用auc。我们怎样才能mlr每次评估所有测试样本的度量而不是唯一的重新采样?

1 个答案:

答案 0 :(得分:1)

您可以对内循环使用不同的重采样策略,然后使用auc

library(mlr)    

ps = makeParamSet(
  makeNumericLearnerParam(id = "cp", default = 0.01, lower = 0, upper = 1)
)
ctrl = makeTuneControlRandom(maxit = 10)
inner = makeResampleDesc("Subsample")
lrn = makeLearner("classif.rpart", predict.type = "prob")
tune = makeTuneWrapper(lrn, resampling = inner, par.set = ps, control = ctrl, measure = auc)

outer = makeResampleDesc("LOO")
r = resample(tune, bc.task, resampling = outer, extract = getTuneResult, measure = auc)

您还可以获取重新采样结果并在其上计算任意性能指标,例如: performance(r$pred, auc)