我想从spark checkpoint目录中读取值 检查点是否仅将数据存储在HDFS中? 我想检查检查点中是否存在数据。我正在使用我的本地机器运行Spark并测试以理解这个概念。
public static JavaStreamingContext createContext(){
SparkConfsparkConf = new SparkConf().setAppName("SparkStreaming");
sparkConf.setMaster("local[2]");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(20));
jssc.checkpoint("C:\\Users\\Desktop\\test");
JavaDStream<String> customReceiverStream = jssc.receiverStream(new
JavaCustomReceiver(MYSQL_DRIVER,
MYSQL_CONNECTION_URL,MYSQL_USERNAME,MYSQL_PWD));
return jssc;
}
public static void main(String[] args) throws InterruptedException {
Function0<JavaStreamingContext> createContextFunc = new Function0<JavaStreamingContext>() {
@Override
public JavaStreamingContext call() {
return createContext();
}
};
JavaStreamingContext streamingContext = JavaStreamingContext.getOrCreate("C:\\Users\\dhala\\Desktop\\test", createContextFunc);
System.out.println(streamingContext.toString());
System.out.println(streamingContext.sparkContext().getCheckpointDir());
streamingContext.start();
streamingContext.awaitTermination();
我想从检查点dir读取..如何找到存储在检查点中的实际值