Question

我正在尝试在pyspark中的kaggle数据集上实现Gradient boost算法以用于学习目的。我面临下面的错误

Traceback (most recent call last):
  File "C:/SparkCourse/Gradientboost.py", line 29, in <module>
    output=assembler.transform(data)
  File "C:\spark\python\lib\pyspark.zip\pyspark\ml\base.py", line 105, in transform
  File "C:\spark\python\lib\pyspark.zip\pyspark\ml\wrapper.py", line 281, in _transform
AttributeError: 'OneHotEncoder' object has no attribute '_jdf'

相应的代码是

from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer,VectorIndexer,OneHotEncoder,VectorAssembler





spark=SparkSession.builder.config("spark.sql.warehouse.dir", "file:///C:/temp").appName("Gradientboostapp").enableHiveSupport().getOrCreate()
data= spark.read.csv("C:/Users/codemen/Desktop/Timeseries Analytics/liver_patient.csv",header=True, inferSchema=True)
#data.show()
print(data.count())
#data.printSchema()
print("After deleting  null  values")

data=data.na.drop()
print(data.count())

data=StringIndexer(inputCol="Gender",outputCol="GenderIndex").fit(data)


#let onehot encode the data

data=OneHotEncoder(inputCol="GenderIndex",outputCol="gendervec")


usedfeature=["Age","gendervec","Total_Bilirubin","Direct_Bilirubin","Alkaline_Phosphotase","Alamine_Aminotransferase","Aspartate_Aminotransferase","Total_Protiens","Albumin","Albumin_and_Globulin_Ratio"]
#
assembler=VectorAssembler(inputCols=usedfeature,outputCol="features")
output=assembler.transform(data)
output.select("features","category").show()

我已经使用String indexer将Gender类别转换为数字形式，然后我尝试在Genderindex值上执行OnehotEncoding。我在代码中执行了VectorAssembler时收到错误。我可以在这里错过非常愚蠢的概念。请帮我搞清楚

Answer 1

这行代码不正确：data=OneHotEncoder(inputCol="GenderIndex",outputCol="gendervec")。您将data设置为等于OneHotEncoder()对象，而不是转换数据。您需要调用transform来对数据进行编码。它看起来应该是这样的。

encoder=OneHotEncoder(inputCol="GenderIndex",outputCol="gendervec") data = encoder.transform(data)

获取AttributeError：'OneHotEncoder'对象在pyspark中没有属性'_jdf'

1 个答案: