Question

我正在通过数据块在PySpark上使用Logistic回归模型，但我无法获得准确度和召回率。一切正常，我可以得到我的ROC，但是Precision和Recall没有属性或库

lrModel = LogisticRegression()

predictions = bestModel.transform(testData)

# Instantiate metrics object
results = predictions.select(['probability', 'label'])
results_collect = results.collect()
results_list = [(float(i[0][0]), 1.0-float(i[1])) for i in results_collect]
scoreAndLabels = sc.parallelize(results_list)

metrics = MulticlassMetrics(scoreAndLabels)

# Overall statistics
precision = metrics.precision()
recall = metrics.recall()
f1Score = metrics.fMeasure()
print("Summary Stats")
print("Precision = %s" % precision)
print("Recall = %s" % recall)
print("F1 Score = %s" % f1Score)

>>>Summary Stats
>>>Precision = 0.0
>>>Recall = 0.0
>>>F1 Score = 0.0

Answer 1

我能够创建自己的功能。它返回所有内容。我正在使用mllib包中的“ MulticlassMetrics（）”。由于它是 multiclass 类别，因此它会为每个标签计算指标，因此，您必须指定要检索的标签。

### Model Evaluator User Defined Functions
def udfModelEvaluator(dfPredictions, labelColumn='label'):

    colSelect = dfPredictions.select(
      [F.col('prediction').cast(DoubleType())
       ,F.col(labelColumn).cast(DoubleType()).alias('label')])

    metrics = MulticlassMetrics(colSelect.rdd)

    mAccuracy = metrics.accuracy
    mPrecision = metrics.precision(1)
    mRecall = metrics.recall(1)
    mF1 = metrics.fMeasure(1.0, 1.0)

    mMatrix = metrics.confusionMatrix().toArray().astype(int)    

    mTP = metrics.confusionMatrix().toArray()[1][1]
    mTN = metrics.confusionMatrix().toArray()[0][0]
    mFP = metrics.confusionMatrix().toArray()[0][1]
    mFN = metrics.confusionMatrix().toArray()[1][0]

    mResults = [mAccuracy, mPrecision, mRecall, mF1, mMatrix, mTP, mTN, mFP, mFN, "Return [[0]=Accuracy, [1]=Precision, [2]=Recall, [3]=F1, [4]=ConfusionMatrix, [5]=TP, [6]=TN, [7]=FP, [8]=FN]"]

    return mResults

调用该函数：

metricsList = udfModelEvaluator(predictionsData, "label")
metricsList

如何使用PySpark测量精度并调用Logistic回归？

1 个答案: