Spark推荐ALS nullpointer异常

时间:2017-04-13 19:27:30

标签: scala apache-spark-mllib recommendation-engine

我正在尝试使用Spark.mllib.recommendation构建一个ALS模型。 我得到一个空指针异常。 我在我使用的列中没有看到任何空值。需要帮助。

import org.apache.spark.mllib.recommendation.Rating
import org.apache.spark.ml.recommendation.ALS

val path = "DataPath"
val data = spark.read.json(path)

data.printSchema()

data.createOrReplaceTempView("reviews")

val raw_reviews = spark.sql("Select reviewerID, cast(asin as int) as ProductID,  overall from reviews")

raw_reviews.printSchema()

import org.apache.spark.ml.feature.StringIndexer


val stringindexer = new StringIndexer()
      .setInputCol("reviewerID")
      .setOutputCol("userID")
  val modelc = stringindexer.fit(raw_reviews)
  val  df = modelc.transform(raw_reviews)

val Array(training,test) = df.randomSplit(Array(0.8,0.2))

val als = new ALS().setMaxIter(5).setRegParam(0.01).setUserCol("userID").setItemCol("ProductID").setRatingCol("overall")

val model = als.fit(training)

org.apache.spark.SparkException:作业因阶段失败而中止:阶段30.0中的任务1失败1次,最近失败:阶段30.0中丢失的任务1.0(TID 94,localhost):java.lang.NullPointerException:索引1处的值为空

0 个答案:

没有答案