Kryo序列化错误火花

时间:2016-10-10 12:14:08

标签: scala hadoop apache-spark

我想注册kryo序列化,因为我已经写了下面的代码,

头等(Select.Class)

var SparkConf = new SparkConf()
      .set("spark.kryo.registrationRequired", "true")
      .set("spark.kryo.registrator", "org.agg.MyRegistrator")

var sc = new SparkContext(SparkConf)
var sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)


val testdata_select = sqlContext.sql("select a,b, from test")

第二类(MyRegistrator.Class)

class MyRegistrator extends KryoRegistrator {
  override def registerClasses(kryo: Kryo) {
    kryo.register(classOf[TransformationCode])
  } 
}

第三类(TransformationCode.Class)

//Transformation Code

但它给了我错误

  

用户类抛出异常:org.apache.spark.SparkException: Task not serializable

0 个答案:

没有答案