Spark插入到hive -JavaPairDStream

Question

Spark插入到hive中：

  finalRDDStream.transform(new Function<JavaPairRDD<OutcomeKey,Tuple2<String,String>>, JavaRDD<Joinkey>>() {

                    private static final long serialVersionUID = 1L;

                    @Override
                    public JavaRDD<Joinkey> call(JavaPairRDD<OutcomeKey, Tuple2<String, String>> arg0)  throws Exception {


                        JavaRDD<Joinkey> a = arg0.map(x -> {
                            OutcomeKey key = x._1;
                            Joinkey joinkey= new Joinkey();     
                            joinkey.setDate(key.getDate().toString());
                            joinkey.setPublicIP(key.getPublicIP());
                            joinkey.setPrivateIP(key.getPrivateIP());
                            joinkey.setMsisdn(x._2._1);
                            joinkey.setUrlVisited(x._2._2);
                            return joinkey;
                            });


                        DataFrame fileDF = sqlContext.createDataFrame(a, Joinkey.class);
                        fileDF .insertInto("test.fileDF ");
                        fileDF .show();
                        return null;

                    }
                });  

我正在尝试将JoinKey类对象插入到hive中。这样我就会收到错误。

16/06/03 03:31:42 ERROR streaming.StreamingContext: Error starting the context, marking it as stopped
java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
        at scala.Predef$.require(Predef.scala:233)
        at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:161)

是否有其他方法可以插入配置单元。

Answer 1

尝试这种方式.....这将插入配置单元

JavaDStream<Joinkey> p = finalRDDStream.transform(new Function<JavaPairRDD<OutcomeKey,Tuple2<String,String>>, JavaRDD<Joinkey>>() {

                    private static final long serialVersionUID = 1L;

                    @Override
                    public JavaRDD<Joinkey> call(JavaPairRDD<OutcomeKey, Tuple2<String, String>> arg0)  throws Exception {


                        JavaRDD<Joinkey> a = arg0.map(x -> {
                            OutcomeKey key = x._1;
                            Joinkey joinkey= new Joinkey();     
                            joinkey.setDate(key.getDate().toString());
                            joinkey.setPublicIP(key.getPublicIP());
                            joinkey.setPrivateIP(key.getPrivateIP());
                            joinkey.setMsisdn(x._2._1);
                            joinkey.setUrlVisited(x._2._2);
                            return joinkey;
                            });
                         a.saveAsTextFile("file:////home/aman/temp");


                        DataFrame fileDF= sqlContext.createDataFrame(a, Joinkey.class);
                        fileDF.insertInto("test.raduisfile");
                        fileDF.show();
                        return null;

                    }
                });
             System.out.println(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>");
             p.print();