springboot spark2.3-mllib integrate

时间:2019-03-19 15:02:09

标签: spring-boot apache-spark apache-spark-mllib

SpringBoot 2.0 integrate spark 2.3 error: factory already defined

Project write with SpringBoot 2.0. In project initialization load spark-mllib model with spark local mode. then stop the sparkSession.

    public static void main(String[] args) {

    // load model 
    ModelUtils.initModel();

    ac = SpringApplication.run(ApplicationLauncher.class, args);
    Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
        @Override
        public void run() {
            SpringApplication.exit(ac);
        }
    }));

}

and load model

    val sc = SparkSession.builder().master("local[2]").appName("bayes").getOrCreate()
    // load model in memory 
    val spamNaiveBayesModel: NaiveBayesModel = NaiveBayesModel.load(sc.sparkContext,bayesPath)
    // stop the spark local 
    sc.close()

then in project use model to predict. When start the project it comes error, spark local can load model success, but springBoot start error: factory already defined. can someone encountered this error code?

0 个答案:

没有答案