从Spark检查点目录

时间:2017-08-09 13:19:14

标签: apache-spark spark-streaming

我想从spark checkpoint目录中读取值 检查点是否仅将数据存储在HDFS中? 我想检查检查点中是否存在数据。我正在使用我的本地机器运行Spark并测试以理解这个概念。

    public static JavaStreamingContext createContext(){      
   SparkConfsparkConf = new SparkConf().setAppName("SparkStreaming");   
   sparkConf.setMaster("local[2]");         
   JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(20)); 
   jssc.checkpoint("C:\\Users\\Desktop\\test");       
 JavaDStream<String> customReceiverStream = jssc.receiverStream(new
     JavaCustomReceiver(MYSQL_DRIVER,
     MYSQL_CONNECTION_URL,MYSQL_USERNAME,MYSQL_PWD));

   return jssc;  
}

public static void main(String[] args) throws InterruptedException {
    Function0<JavaStreamingContext> createContextFunc = new Function0<JavaStreamingContext>() {
        @Override
        public JavaStreamingContext call() {
            return createContext();
        }
    };

    JavaStreamingContext streamingContext = JavaStreamingContext.getOrCreate("C:\\Users\\dhala\\Desktop\\test", createContextFunc);
    System.out.println(streamingContext.toString());
    System.out.println(streamingContext.sparkContext().getCheckpointDir());
    streamingContext.start();
    streamingContext.awaitTermination();

我想从检查点dir读取..如何找到存储在检查点中的实际值

0 个答案:

没有答案