如何在JAVA 8中爆炸Spark DataFrame(sql)并遍历Rows

时间:2016-10-14 10:17:15

标签: java scala apache-spark dataframe spark-dataframe

我试图在Java中复制一些在Scala中很容易实现的东西。爆炸Spark DataFrame并遍历行以应用逻辑(并返回修改的行)

Scala版本示例(工作案例)

case class someModel(A: String, B: String)
someDF.explode(someDF("A"), someDF("B")) {
      case Row(A: String, B: String) =>

          List(someModel("a value", "a second value"))

    }

Java版本(我到达了多远,没有工作)

someDF.explode(columnsToExplodeSeq, new Function1<Row, TraversableOnce<Row>>() {
            @Override
            public TraversableOnce<Row> apply(Row v1) {
                return null;
            }

            @Override
            public <A> Function1<A, TraversableOnce<Row>> compose(Function1<A, Row> g) {
                return super.compose(g); // this give compile error, not calling super method!
            }

            @Override
            public <A> Function1<Row, A> andThen(Function1<TraversableOnce<Row>, A> g) {
                return super.andThen(g); // this give compile error, not calling super method!
            }
});

0 个答案:

没有答案