Question

我正在做一个pyspark ml样本练习，我需要存储一个模型并将其读回。我能够成功保存模型，但是当我尝试读取/加载回模型时，它抛出异常。我是火花ml和python的新手，请指导我。

代码：

from pyspark.sql import *
from pyspark.ml.feature import RFormula
from pyspark.ml.classification import LogisticRegression
from pyspark.ml import Pipeline
from pyspark.ml.tuning import ParamGridBuilder
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.tuning import TrainValidationSplit
from pyspark.ml.tuning import TrainValidationSplitModel


spark = SparkSession.builder.appName("LocalMLSparkSession").master("local").getOrCreate()

df = spark.read.json("/to_data/simpleml.json").orderBy("value2")

df.select(df.color).distinct().show(10, False)

train, test = df.randomSplit([0.7, 0.3])


rForm = RFormula()
ls = LogisticRegression().setLabelCol("label").setFeaturesCol("features")

# setting pipeline
stages = [rForm,ls]
pipeline = Pipeline().setStages(stages)


#setting param grid builder
params = ParamGridBuilder()\
     .addGrid(rForm.formula,["lab ~ . + color:value1", "lab ~ . + color:value1 + color:value2"])\
     .addGrid(ls.elasticNetParam, [0.0, 0.5, 1.0])\
     .addGrid(ls.regParam,[0.1, 0.2])\
     .build()

#setting evaluator
evaluator = BinaryClassificationEvaluator()\
            .setMetricName("areaUnderROC")\
            .setRawPredictionCol("prediction")\
            .setLabelCol("label")

#checking hyperparameters to train datasets
tvs = TrainValidationSplit()\
    .setTrainRatio(0.75)\
    .setEstimatorParamMaps(params)\
    .setEstimator(pipeline)\
    .setEvaluator(evaluator)


tvsFitted = tvs.fit(train)

evl = evaluator.evaluate(tvsFitted.transform(test))

tvsFitted.transform(test).select("features", "label", "prediction").show(10,False)

print(evl)
pip_model = tvsFitted.bestModel
pip_model.write().overwrite().save("/to_path/sparkml/model")

model = TrainValidationSplitModel().load("/to_path/sparkml/model")
model.transform(test)

例外：

Traceback (most recent call last):
  File "/home/dd/dd/python-workspace/SparkMLPipelineDemo.py", line 59, in <module>
    model = TrainValidationSplitModel().load("/to_path/sparkml/model")
TypeError: __init__() missing 1 required positional argument: 'bestModel'

Process finished with exit code 1

Answer 1

您需要在加载时删除括号，例如替换：

model = TrainValidationSplitModel().load("/to_path/sparkml/model")

与

model = TrainValidationSplitModel.load("/to_path/sparkml/model")

pyspark-2.3 sparkml LogisticRegression模型加载问题

1 个答案: