Question

我正在尝试使用BinaryclassificationMetrics计算精度，通过阈值调用LogisticRegressionwithLBFGS。我得到了所有这些。我试图找出是否可以获得PR和AUC曲线的图形输出。

在下面粘贴我的代码：

import org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
import org.apache.spark.mllib.evaluation.{BinaryClassificationMetrics, MulticlassMetrics}
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}



object log_reg_eval_metric {

  def main(args: Array[String]): Unit = {


    System.setProperty("hadoop.home.dir", "c:\\winutil\\")


    val sc = new SparkContext(new SparkConf().setAppName("SparkTest").setMaster("local[*]"))

    val sqlContext = new org.apache.spark.sql.SQLContext(sc);

    val data: RDD[String] = sc.textFile("C:/Users/user/Documents/spark-1.5.1-bin-hadoop2.4/data/mllib/credit_approval_2_attr.csv")


    val parsedData = data.map { line =>
      val parts = line.split(',').map(_.toDouble)
      LabeledPoint(parts(0), Vectors.dense(parts.tail))
    }

    //Splitting the data
    val splits: Array[RDD[LabeledPoint]] = parsedData.randomSplit(Array(0.7, 0.3), seed = 11L)
    val training: RDD[LabeledPoint] = splits(0).cache()
    val test: RDD[LabeledPoint] = splits(1)



    // Run training algorithm to build the model
    val model = new LogisticRegressionWithLBFGS()
      .setNumClasses(2)
      .run(training)
    // Clear the prediction threshold so the model will return probabilities
    model.clearThreshold

    // Compute raw scores on the test set
    val predictionAndLabels = test.map { case LabeledPoint(label, features) =>
      val prediction = model.predict(features)
      (prediction, label)
    }

    // Instantiate metrics object
    val metrics = new BinaryClassificationMetrics(predictionAndLabels)

    // Precision by threshold
    val precision = metrics.precisionByThreshold
    precision.foreach { case (t, p) =>
      println(s"Threshold: $t, Precision: $p")
    }


    // Precision-Recall Curve
    val PRC = metrics.pr

    print(PRC)



  }
}

print（PRC）的输出：

UnionRDD[39] at union at BinaryClassificationMetrics.scala:108

我不确定什么是联合RDD以及如何使用它。有没有其他方法来获得图形输出。做我的研究。任何建议都会很棒。

Answer 1

您可以使用spark.ml包中的BinaryLogisticRegressionTrainingSummary。它提供PR和ROC值作为数据帧。

您可以将这些值输入到任何渲染实用程序以查看特定曲线。（任何带有x和y值的多线图将显示曲线。）

Spark中二进制分类的评估指标：AUC和PR曲线

1 个答案: