Question

我正在开发spark（v2.4.3）的一部分，该部分使用es-hadoop（v7.3.1）在spring-boot 2（web）上读取elasticsearch（v7.3.0）。

**参考https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html

当我使用“ mvn spring-boot：run”运行它时，它工作正常。

但是，当我使用“ java -jar output.jar”运行它时，JavaEsSpark在访问elasticsearch时出错。 我有很多想法，但我不知道为什么。

“ ExceptionInInitializerError：null .. RestService” “无法实例化sql.DefaultSource15” “ URI不是分层的..版本。”

我寻求您的帮助（-.-）

** ps。我在jar文件中检查了软件包。 =>好。我尝试了SparkSession，但也失败了。（java -jar执行）

**我的代码 https://github.com/maxmin93/agens-spark/blob/master/src/main/java/net/bitnine/agensspark/AgenssparkApplication.java

LOGGER.info("SpringBootSpark To Elasticsearch Application: {}, {}, {}, {}"
                , appName, nodes, port, sparkHome);

String master = "local[*]";
SparkConf conf = new SparkConf()
        .setAppName(appName)
        .setSparkHome(sparkHome)
        .setMaster(master)
                .set("spark.executor.memory", "2g")
        .set("spark.driver.memory", "2g")
        .set("spark.eventLog.enabled","false")
        .set("es.nodes.wan.only", "true")
        .set("es.nodes", nodes)
        .set("es.port", port)
        .set("es.mapping.id", "id")
        .set("es.write.operation", "upsert")
        .set("es.index.read.missing.as.empty", "true");

        JavaSparkContext jsc = new JavaSparkContext(conf);

// for DEBUG
System.out.println("\n\n==================================\n");
System.out.println(Arrays.stream(jsc.getConf().getAll()).map(t->{
    return t._1()+"="+t._2();
}).collect(Collectors.joining(", ")));

JavaPairRDD<String, Map<String, Object>> esRDD =
    JavaEsSpark.esRDD(jsc, resource, "?q=datasource:sample");
System.out.println("\n**count = "+esRDD.count());

错误消息

==================================

spark.app.id=local-1568944125286, spark.executor.memory=2g, spark.home=/Users/bgmin/Servers/spark, es.port=15619, spark.app.name=agensspark, spark.executor.id=driver, es.mapping.id=id, es.index.read.missing.as.empty=true, spark.driver.host=192.168.0.20, spark.driver.memory=2g, spark.driver.port=49672, spark.master=local[*], es.write.operation=upsert, es.nodes=27.117.163.21, spark.eventLog.enabled=false, es.nodes.wan.only=true
2019-09-20 10:48:45.397  INFO 14660 --- [           main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-09-20 10:48:45.401 ERROR 14660 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.ExceptionInInitializerError: null
    at org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:216) ~[elasticsearch-hadoop-7.3.2.jar!/:7.3.2]
    at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions$lzycompute(AbstractEsRDD.scala:79) ~[elasticsearch-hadoop-7.3.2.jar!/:7.3.2]
    at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions(AbstractEsRDD.scala:78) ~[elasticsearch-hadoop-7.3.2.jar!/:7.3.2]

执行java -jar时，JavaEsSpark.esRDD失败，并出现ExceptionInInitializerError，URI不分层

0 个答案: