我正在使用spark 2.0并尝试将Java RDD转换为数据帧。
这是我使用它的代码,我的bean有嵌套的bean。
JavaRDD<XXX> mappedRDD = hbaseRDD.map(s->{
//final long serialVersionUID = -2021713021648730786L;
XXX xx=new XXX();
return xx;
});
Dataset<Row> df = sparkSession.createDataFrame(mappedRDD, XXX.class);
我收到以下错误
Exception in thread "main" java.lang.StackOverflowError
at sun.reflect.generics.repository.GenericDeclRepository.getTypeParameters(GenericDeclRepository.java:84)
at java.lang.Class.getTypeParameters(Class.java:715)
at org.spark_project.guava.reflect.Types$ParameterizedTypeImpl.<init>(Types.java:288)
at org.spark_project.guava.reflect.Types.newParameterizedType(Types.java:98)
at org.spark_project.guava.reflect.TypeToken.toGenericType(TypeToken.java:917)
at org.spark_project.guava.reflect.TypeToken.getSupertype(TypeToken.java:401)
at org.apache.spark.sql.catalyst.JavaTypeInference$.elementType(JavaTypeInference.scala:132)
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:101)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:117)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:115)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:115)
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:101)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:117)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:115)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)