Question

我有一个火花流应用程序，它从kafka主题中提取数据，然后将该数据与维度查找相关联。虽然我已经设法在第一次运行时从hive加载维度查找表到RDD;我希望这个维度lkp rdd每隔一小时刷新一次。

根据我的理解，SparkStreaming实际上是一个spark调度程序，我想知道是否可以创建Dimensional查找RDD的JavaDStream，并使用Spark Streaming在预定的时间间隔内刷新它。问题是我不知道如何处理这个问题，根据我的理解，RDD是不可变的，这意味着甚至可以在spark中刷新JavaDStream并将其与运行在不同计划中的JavaDStream连接起来？

当前代码：

    System.out.println("Loading from Hive Tables...");

//Retrieve Dimensional LKP data from hive table into RDD (CombineIVAPP function retrieves the data from Hive and performs initial joins)
final JavaPairRDD<String, Tuple2<modelService,modelONT>> LKP_IVAPP_DIM = CombineIVAPP();
LKP_IVAPP_DIM.cache();

System.out.println("Mapped Tables to K/V Pairs");

    //Kafka Topic settings
    Map<String, Integer> topicMap = new HashMap<String, Integer>();
    topicMap.put(KAFKA_TOPIC,KAFKA_PARA);

    //Begin to stream from Kafka Topic
    JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(
            jssc, ZOOKEEPER_URL, KAFKA_GROUPID, topicMap);


    //Map messages from Kafka Stream to Tuple
    JavaDStream<String> json = messages.map(
            new Function<Tuple2<String, String>, String>() {
                @Override
                public String call(Tuple2<String, String> message) {
                    return message._2();
                }
            }
    );

    //Map kafka JSON string to K/V RDD
    JavaPairDStream<String, modelAlarms> RDD_ALARMS = json.mapToPair(new KafkaToRDD());

    //Remove the null values
    JavaPairDStream<String, modelAlarms> RDD_ALARMS_FILTERED = RDD_ALARMS.filter(new Function<Tuple2<String, modelAlarms>, Boolean>() {
                @Override
                public Boolean call(Tuple2<String, modelAlarms> item) {
                    return item != null;
                }
            });

    //Join Alarm data from Kafka topic with hive lkp table LKP_IVAPP_DIM
    JavaPairDStream<String, Tuple2<modelAlarms,Tuple2<modelService,modelONT>>> RDD_ALARMS_JOINED = RDD_ALARMS_FILTERED.transformToPair(new Function<JavaPairRDD<String, modelAlarms>, JavaPairRDD<String, Tuple2<modelAlarms, Tuple2<modelService,modelONT>>>>() {
        @Override
        public JavaPairRDD<String, Tuple2<modelAlarms, Tuple2<modelService,modelONT>>> call(JavaPairRDD<String, modelAlarms> v1) throws Exception {
            return v1.join(LKP_IVAPP_DIM);
        }
    });

通过Spark Streaming刷新RDD LKP表

0 个答案: