从班级制作RDD

时间:2019-06-18 09:22:48

标签: scala apache-spark rdd

如果我有List并执行所需的操作(如

),则可以创建RDD
val li = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8)
val rdd = sc.parallelize(li, 5)
val rddm =rdd.map(x =>x * 2).map(println(_)).collect()

但是当我尝试从一个类创建RDD时,它给出错误,指出对象不可序列化(类:BAT1,值:BAT1 @ 78f84d5)。我的代码是

object sparkBAT {
 def main(args: Array[String]): Unit = {
 val N = 10
 val d = 5
 val MinVal = -10
 val MaxVal =  10
 val conf = new  SparkConf().setMaster(locally("local")).setAppName("spark Demo")
 val sc = new SparkContext(conf)

 val ba = List.fill(N)(new BAT(d, MinVal, MaxVal))

 val rdd = sc.parallelize(ba, 5)
 rdd.map(x=> (x.BestFitness ,(x.fitness,x.position))).take(5)
 rdd.mapPartitions( y => y.map{x => println(x.position, x.fitness)})
 }
}
class BAT ( dim:Int  ,  min:Double  ,  max:Double){
val random = new Random()
var position      : List[Double]      =   List.fill(dim) (random.nextDouble() * (max-min)+min )
var velocity      :List[Double]       =   List.fill(dim)( math.random)
var PulseRate     : Double            =   0.1
var LoudnessRate  :Double             =   0.95
var frequency     :Double             =   math.random
var fitness       :Double             =   math.random
var BestPosition  :List[Double]       =   List.fill(dim)(math.random)
var BestFitness   :Double             =   math.random 
}

如何从 BAT 类创建RDD?

0 个答案:

没有答案