Question

我正在使用以下命令以集群模式提交spark作业，并使用-files

显式传递了 stage.cde.properties 文件

spark-submit --class com.cse.cde.CassandraHiveLoader \
--master yarn --deploy-mode cluster \
--files /opt/mapr/spark/spark-2.2.1/conf/hive-site.xml,stage.cde.properties \
--jars $(echo lib/*.jar | tr ' ' ',') \
--driver-memory 7g \
--conf 'spark.driver.extraClassPath=lib/guava-19.0.jar' \
--conf 'spark.executor.extraClassPath=lib/guava-19.0.jar' \
--conf 'spark.executor.memory=8g' \
--conf 'spark.executor.instances=50' \
--conf 'spark.yarn.executor.memoryOverhead=1500'  \
--conf 'spark.cassandra.input.consistency.level=ONE' \
--conf 'spark.cassandra.output.consistency.level=LOCAL_QUORUM' \
--conf 'spark.shuffle.service.enabled=false' \
--conf 'spark.dynamicAllocation.enabled=false' \
--conf 'spark.yarn.executor.memoryOverhead=1024' \
--conf 'spark.local.dir=/users/kaka/users/prsahoo2/spark/local' \
--conf 'spark.log.dir=/users/kaka/users/prsahoo2/spark/log'  cde-loader.jar stage

并使用以下代码加载 stage.cde.properties 配置文件。

public static Config loadConf() {
        String env = System.getenv(SYSTEM_ENV_CONFIG_TYPE);
        LOGGER.info("Loading configurations for :"+env);
        System.out.println("Loading configurations for :"+env);
        ConfigType cfgType = ConfigType.valueOf(null == env ? ConfigType.dev.name() : env);
        nonNull(cfgType, "Config type for loading configs cannot be null");
        String cfgName = cfgType.name() + ".cde.properties";
        LOGGER.info("Loading configurations present in: {}", cfgName);
        System.out.println("Loading configurations present in: " + cfgName);
        Config configs = ConfigFactory.load(ConfigFactory.parseFile(new java.io.File(cfgName)));
        LOGGER.info("Loaded configs: {}", configs);
        return configs;
    }

错误：

LogType:stderr
Log Upload Time:Thu Sep 13 23:01:47 -0700 2018
LogLength:897
Log Contents:
com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'app'
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:138)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
    at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:197)
    at com.cisco.cse.cdf.CassandraToHive.main(CassandraHiveLoader.java:73)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)
End of LogType:stderr

LogType:stdout
Log Upload Time:Thu Sep 13 23:01:47 -0700 2018
LogLength:0
Log Contents:
End of LogType:stdout

在提交作业后，由于 stage.cde.properties 文件对于群集模式下的驱动程序不可用，因此出现以上错误消息。有人可以帮我吗？

收到错误消息com.typesafe.config.ConfigException $ Missing：找不到关键应用程序的配置设置

0 个答案: