如果我有List并执行所需的操作(如
),则可以创建RDDval li = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8)
val rdd = sc.parallelize(li, 5)
val rddm =rdd.map(x =>x * 2).map(println(_)).collect()
但是当我尝试从一个类创建RDD时,它给出错误,指出对象不可序列化(类:BAT1,值:BAT1 @ 78f84d5)。我的代码是
object sparkBAT {
def main(args: Array[String]): Unit = {
val N = 10
val d = 5
val MinVal = -10
val MaxVal = 10
val conf = new SparkConf().setMaster(locally("local")).setAppName("spark Demo")
val sc = new SparkContext(conf)
val ba = List.fill(N)(new BAT(d, MinVal, MaxVal))
val rdd = sc.parallelize(ba, 5)
rdd.map(x=> (x.BestFitness ,(x.fitness,x.position))).take(5)
rdd.mapPartitions( y => y.map{x => println(x.position, x.fitness)})
}
}
class BAT ( dim:Int , min:Double , max:Double){
val random = new Random()
var position : List[Double] = List.fill(dim) (random.nextDouble() * (max-min)+min )
var velocity :List[Double] = List.fill(dim)( math.random)
var PulseRate : Double = 0.1
var LoudnessRate :Double = 0.95
var frequency :Double = math.random
var fitness :Double = math.random
var BestPosition :List[Double] = List.fill(dim)(math.random)
var BestFitness :Double = math.random
}
如何从 BAT 类创建RDD?