我想注册kryo序列化,因为我已经写了下面的代码,
头等(Select.Class)
var SparkConf = new SparkConf()
.set("spark.kryo.registrationRequired", "true")
.set("spark.kryo.registrator", "org.agg.MyRegistrator")
var sc = new SparkContext(SparkConf)
var sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
val testdata_select = sqlContext.sql("select a,b, from test")
第二类(MyRegistrator.Class)
class MyRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[TransformationCode])
}
}
第三类(TransformationCode.Class)
//Transformation Code
但它给了我错误
用户类抛出异常:
org.apache.spark.SparkException: Task not serializable