Question

我正在使用h2o软件包对GBM进行二进制分类。我想评估某个变量的预测能力，如果我是对的，我可以通过比较具有特定变量的模型的AUC和不具有特定变量的模型的AUC来进行评估。

我以泰坦尼克号数据集为例。

所以我的假设是：年龄对于某人能否生存具有重要的预测价值。

df <- h2o.importFile(path = "http://s3.amazonaws.com/h2o-public-test-data/smalldata/gbm_test/titanic.csv")
response <- "survived"
df[[response]] <- as.factor(df[[response]])           
## use all other columns (except for the name) as predictors
predictorsA <- setdiff(names(df), c(response, "name")) 
predictorsB <- setdiff(names(df), c(response, "name", "age")) 

splits <- h2o.splitFrame(
  data = df, 
  ratios = c(0.6,0.2),   ## only need to specify 2 fractions, the 3rd is implied
  destination_frames = c("train.hex", "valid.hex", "test.hex"), seed = 1234
)
train <- splits[[1]]
valid <- splits[[2]]
test  <- splits[[3]]

gbmA <- h2o.gbm(x = predictorsA, y = response, distribution="bernoulli", training_frame = train)
gbmB <- h2o.gbm(x = predictorsB, y = response, distribution="bernoulli", training_frame = train)

## Get the AUC
h2o.auc(h2o.performance(gbmA, newdata = valid))
[1] 0.9631624
h2o.auc(h2o.performance(gbmB, newdata = test))
[1] 0.9603211

我知道pROC软件包具有 roc.test 函数，用于比较两条ROC曲线的AUC，我希望将此函数应用于我的h2o模型的结果。

Answer 1

您可以执行以下操作-

valid_A <- as.data.frame(h2o.predict(gbmA,valid))
valid_B <- as.data.frame(h2o.predict(gbmB,valid))
valid_df <- as.data.frame(valid)
roc1 <- roc(valid_df$survived,valid_A$p1)
roc2 <- roc(valid_df$survived,valid_B$p1)


> roc.test(roc1,roc2)

    DeLong's test for two correlated ROC curves

data:  roc1 and roc2
Z = -0.087489, p-value = 0.9303
alternative hypothesis: true difference in AUC is not equal to 0
sample estimates:
AUC of roc1 AUC of roc2 
  0.9500141   0.9504367

将pROC软件包与h2o一起使用

1 个答案: