我正在使用以下代码启动spark作业:
public static void main(String[] args) throws InterruptedException, ExecutionException {
Process sparkProcess;
try {
sparkProcess = new SparkLauncher()
.setSparkHome("C:\\spark-2.0.0-bin-hadoop2.7")
.setAppResource("hdfs://server:9000/inputs/test.jar")
.setMainClass("com.test.TestJob")
.setMaster("spark://server:6066") //rest URL of spark
.setVerbose(true)
.setDeployMode("cluster")
.addAppArgs("abc")
.launch();
} catch (IOException e) {
throw new RuntimeException(e);
}
ExecutorService executorService = Executors.newSingleThreadExecutor();
Future<Integer> submit = executorService.submit(() -> {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(sparkProcess.getInputStream()))) {
while (sparkProcess.isAlive()) {
try {
System.out.println("input stream line:" + reader.readLine());
Thread.sleep(1000);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
return sparkProcess.exitValue();
});
System.out.println("Exit value:" + submit.get());
sparkProcess.waitFor();
}
我的test.jar文件代码如下:
public static void main(String[] args) {
SparkSession spark = SparkSession.builder().appName("test").config("spark.executor.memory", "1g")
.config("spark.executor.cores", 1).getOrCreate();
JavaSparkContext context = new JavaSparkContext(spark.sparkContext());
long count = context.parallelize(Arrays.asList(1, 2, 3, 4, 5)).count();
System.out.println("Count:" + count); //want to retrieve this detail on launching application.
spark.stop();
}
我想在成功完成spark工作时启动app时检索计数。但是从InputStream
sparkProcess
开始,我一直在接收null
我在这里做错了什么?