我在SPARK SQL代码中看到一个问题,line3永远不会被执行,即使line1有很多行,我的意思是dataFrame2中有行,而且方法call()
永远不会在JavaRDD上执行。请指教,我无法解决这个问题。
listOfDStreams.foreachRDD(new Function<JavaRDD<String>,Void>(){
@Override
public Void call(JavaRDD<String> rdd) throws Exception {
printToLogsAndConsole("listOfDStreams.foreachRDD","listOfDStreams.foreachRDD");
DataFrame dataFrame=sqlContext.read().json(rdd);
DataFrame dataFrame2=null;
if(dataFrame!=null && rdd!=null&&!rdd.isEmpty()){
dataFrame.registerTempTable("test");
dataFrame.groupBy("dst").count().show();
dataFrame2= sqlContext.sql("select * from test");
List<Row> rows=dataFrame2.collectAsList(); //line 1
dataFrame2.toJavaRDD().map(new Function<Row, String>() { //line2
@Override
public String call(Row arg0) throws Exception {
// TODO Auto-generated method stub
printToLogsAndConsole("in java rdd ","in java rdd");//line3
return null;
}
});
}
return null;
}
});