Question

当我从命令行提交spark作业时，作业将出现在spark历史服务器UI和hadoop资源管理器UI上，但是当我从java代码提交时，作业显示在hadoop资源管理器UI上但不显示在spark历史记录中服务器用户界面。

火花程序是一样的，只有我提交它的方式不是。

提交代码：

public class submitJava {
    public static Map getPropXmlAsMap(String path) throws Exception, IOException{
        File file = new File(path);
        SAXBuilder builder = new SAXBuilder();
        Document doc = builder.build(file);
        Element root = doc.getRootElement();
        List props = root.getChildren("property");
        Map result = new HashMap();
        for(Iterator iter = props.iterator(); iter.hasNext();){
            Element element = (Element) iter.next();
            Element nameele = element.getChild("name");
            Element valueele = element.getChild("value");
            String name = nameele.getText();
            String value = valueele.getText();
            result.put(name, value);
        }
        return result;

    }
    public static void fillProperties(Configuration conf,Map map){
        Iterator iter = map.entrySet().iterator();
        while(iter.hasNext()){
            Map.Entry entry = (Map.Entry)iter.next();
            String key = (String)entry.getKey();
            String value = (String)entry.getValue();
            conf.set(key, value);
        }
    }
    public static void main(String[] args) {
        Configuration config = new Configuration();

        SparkConf sparkConf = new SparkConf();

        sparkConf.setMaster("yarn-cluster");


        try {
            fillProperties(config,getPropXmlAsMap("/home/tseg/hadoop-2.6.0/etc/hadoop/core-site.xml"));
            fillProperties(config,getPropXmlAsMap("/home/tseg/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));

        } catch (Exception e) {

            e.printStackTrace();
        }
        List<String> runArgs = Arrays.asList(
                "--class","org.apache.spark.examples.JavaSparkPi",
                "--addJars","hdfs://tseg0:9010/user/tseg/hh/spark-assembly-1.4.0-hadoop2.6.0.jar",
                "--jar","file:////home/tseg/hh/spark-1.4.0-bin-hadoop2.6.0/lib/spark-examples-1.4.0-hadoop2.6.0.jar"
                ,"--arg","10"
                );

        System.setProperty("SPARK_YARN_MODE", "true");

        ClientArguments argss = new ClientArguments(runArgs.toArray(new String[runArgs.size()]),sparkConf);

        Client client = new Client(argss,config,sparkConf);

        ApplicationId applicationId = client.submitApplication();


    }


}

Answer 1

查看https://spark.apache.org/docs/1.2.0/running-on-yarn.html

它声明存在属性GdkRGBA background = {0., 0., 0., .5}; vte_terminal_set_color_background(VTE_TERMINAL(terminal), &background);，默认为空，因为历史服务器是可选服务。从代码提交时可能没有设置，但是在通过命令行启动时设置。

从代码中提交spark程序，并在历史服务器ui上显示该作业

1 个答案: