value mkString is not a value of org.apache.spark.rdd.RDD[Int]

Question

I changed this line:

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}.take(20).mkString("::")

to:

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}.mkString("::") 

But Eclipse is giving me an error hint: value mkString is not a value of org.apache.spark.rdd.RDD[Int]

What does this error mean?

Answer 1

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}

This returns an org.apache.spark.rdd.RDD[Int] which is not GenTraversableOnce. Although it has a lot of methods defined which makes it like a Scala collection of Int, it is not (abstract class RDD[T] extends Serializable with Logging). It's a bit like a promise of a collection Int. You have to poll the collection out before you mkString with the results.

Call .collect() on the RDD[Int] before you perform mkString.

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}.collect.mkString("::")

Or you can add an implicit conversion:

implicit def toArray[T](rdd: RDD[T]) = rdd.collect()