I am using the caret
package to train models for a classification problem. I know that defaultSummary
can be used to calculate Accuracy/Kappa (and SDs), and twoClassSummary
will calculate Sens/Spec. I would like to also calculate positive and negative predictive values (PPV/NPV, and SDs) as easily as possible, in the same fashion. I have come up with a solution, but wonder if anyone could confirm that the solution appears reasonable.
First, I generate the predictive values:
predictiveValues <- function (data, lev = NULL, model = NULL){
PPVobj <- posPredValue(data[, "pred"], data[, "obs"])
NPVobj <- negPredValue(data[, "pred"], data[, "obs"])
out <- c(PPVobj, NPVobj)
names(out) <- c("PPV", "NPV")
out}
Then combine these values into a custom summary, to pass to caret::trainControl
custom.summary <- function(data, lev = NULL, model = NULL){
a <- defaultSummary(data, lev, model)
b <- twoClassSummary(data, lev, model)
c <- predictiveValues(data, lev, model)
out<-c(a,b,c)
out}
My primary question is whether I have overlooked an important error-check that might be missed by twoClassSummary but could invalidate the predictiveValues function. For instance, if there are NAs in the predictions of the model during CV, I might need to include a na.rm somewhere in my additional function. I think it is unclear without developer-level knowledge of caret
.
Alternatively, is there an easier way of doing this that I have overlooked?
Thanks for any help people can provide. Adam