我想使用pyspark进行k倍交叉验证以微调参数,而我正在使用pyspark.ml。 我收到属性错误。
AttributeError:“ DataFrame”对象没有属性“ _jdf”
我最初尝试使用pyspark.mllib,但无法成功执行k倍交叉验证
import pandas as pd
from pyspark import SparkConf, SparkContext
from pyspark.ml.classification import DecisionTreeClassifier
data=pd.read_csv("file:///SparkCourse/wdbc.csv", header=None)
type(data)
print(data)
conf = SparkConf().setMaster("local").setAppName("SparkDecisionTree")
sc = SparkContext(conf = conf)
# Create initial Decision Tree Model
dt = DecisionTreeClassifier(labelCol="label", featuresCol="features",
maxDepth=3)
# Train model with Training Data
dtModel = dt.fit(data)
# I expect the model to be trained but I'm getting the following error
AttributeError: 'DataFrame' object has no attribute '_jdf'
注意:我可以打印数据。 dtModel错误
答案 0 :(得分:2)
将Panadas转换为Spark
from pyspark.sql import SQLContext
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
spark_dff = sqlContext.createDataFrame(panada_df)
答案 1 :(得分:0)
如果指标评估错误,您可能会:
# Spark model, transformed test, converted to pandas df
predictions = model.transform(test)
predDF = predictions.toPandas()
predDF.head()
eval_acc = MulticlassClassificationEvaluator(
labelCol='Label_index',
predictionCol='prediction',
metricName='accuracy'
)
# Evaluate Performance
acc = eval_acc.evaluate(predDF) # Error
print(f"accuracy: {acc}")
我忘记了 predDF是Pandas DataFrame。 需要预测,因为它是Spark数据框。
acc = eval_acc.evaluate(predictions) # Works
print(f"accuracy: {acc}")