Question

我想向纱线群集提交火花作业，我从“https://github.com/mahmoudparsian/data-algorithms-book/blob/master/misc/how-to-submit-spark-job-to-yarn-from-java-code.md”获得参考。

这是我的代码：

package learn.spark;

import org.apache.spark.deploy.yarn.Client;
import org.apache.spark.deploy.yarn.ClientArguments;
import org.apache.hadoop.conf.Configuration;
import org.apache.spark.SparkConf;

public class SubmitJob {

public static void main(String[] arguments) throws Exception {

    String[] args = new String[] {

        "--jar", "lib/spark-examples-1.4.0-hadoop2.6.0.jar",
        "--class", "org.apache.spark.examples.mllib.JavaKMeans",
        "--num-executors", "32",
        "--executor-cores", "4",
        "--executor-memory", "16G", 
        "--driver-memory", "8G",    
        "--addJars", "./lib/spark-assembly-1.4.0-hadoop2.6.0.jar",

        "--arg", "/data/kmeans_data.txt",
        "--arg", "5",
        "--arg", "9"    
    };

    Configuration config = new Configuration();

    System.setProperty("SPARK_YARN_MODE", "true");

    SparkConf sparkConf = new SparkConf();

    ClientArguments cArgs = new ClientArguments(args, sparkConf); 

    Client client = new Client(cArgs, config, sparkConf); 

    client.run(); 
}
}

我编译源码和jar。通过以下cmd运行程序：

java -Xbootclasspath/a:./lib/spark-assembly-1.4.0-hadoop2.6.0.jar: -jar learn.spark.SubmitJob.jar

不幸的是，我收到了以下错误： The output

我无法理解这个问题，或者如何向纱线集群提交火花作业？还有更好的方法吗？

从java代码提交spark作业并获取java.lang.NullPointerException

0 个答案: