Question

该代码适用于默认的1个分区。在为JDBC连接引入基于列的分区之后，作业导致StackOverflow错误。

sparkSession.read
  .option("numPartitions",partitions)
  .option("lowerBound",lowerBound)
    .option("upperBound",upperBound)
  .option("partitionColumn",partitionColumn)
    .jdbc(jdbcConf.getProperty("connection_string"), tableName, jdbcConf)
    .selectExpr(columns: _*)
  .where(whereExpr)

Spark作业因StackOverflow错误退出。

StackTrace：

java.lang.StackOverflowError
at org.apache.spark.sql.catalyst.expressions.Canonicalize$$anonfun$expressionReorder$3.isDefinedAt(Canonicalize.scala:66)
at org.apache.spark.sql.catalyst.expressions.Canonicalize$$anonfun$expressionReorder$3.isDefinedAt(Canonicalize.scala:66)
at org.apache.spark.sql.catalyst.expressions.Canonicalize$.org$apache$spark$sql$catalyst$expressions$Canonicalize$$gatherCommutative(Canonicalize.scala:50)
at org.apache.spark.sql.catalyst.expressions.Canonicalize$$anonfun$org$apache$spark$sql$catalyst$expressions$Canonicalize$$gatherCommutative$1.apply(Canonicalize.scala:50)
at org.apache.spark.sql.catalyst.expressions.Canonicalize$$anonfun$org$apache$spark$sql$catalyst$expressions$Canonicalize$$gatherCommutative$1.apply(Canonicalize.scala:50)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:344)

Spark：在Spark JDBC连接中使用分区后出现Stackoverflow错误

0 个答案: