Apache Spark Scala-Hive插入会引发“太大的数据框错误”

时间：2018-11-27 16:06:26

标签： scala apache-spark hadoop hive apache-spark-sql

我试图使用下面的代码插入Hive，但是由于某种原因，它总是会失败。我尝试调整内存，但没有帮助。

错误堆栈跟踪：

[Stage 4:=====================================================>(999 + 1) / 1000]18/11/27 09:59:44 WARN TaskSetManager: Lost task 364.0 in stage 4.0 (TID 1367, spark-node, executor 1): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.hive.SparkHiveDynamicPartitionWriterContainer.writeToFile(hiveWriterContainers.scala:328)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:159)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:159)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.shuffle.FetchFailedException: Too large frame: 5587345928
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at org.apache.spark.sql.hive.SparkHiveDynamicPartitionWriterContainer.writeToFile(hiveWriterContainers.scala:286)
... 8 more
Caused by: java.lang.IllegalArgumentException: Too large frame: 5587345928
at org.spark_project.guava.base.Preconditions.checkArgument(Preconditions.java:119)
at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:133)
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:81)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
... 1 more

这是我的火花提交： spark-submit --class com.generic.MSSQLHiveIngestion --master yarn --num-executors 8 --executor-cores 2 --executor-memory 16G --driver-memory 8G --driver-cores 4 --conf spark。 yarn.executor.memoryOverhead = 1G data-ingestion.jar

下面是我的sudo代码：

//create spark session first
val spark = SparkSession.builder()
  .appName("MSSQLIngestion")
  .master("yarn")
  .config("spark.sql.caseSensitive", "false")
  .config("spark.sql.shuffle.partitions", "1000")
  .config("spark.shuffle.spill", "true")
  .config("spark.executor.extraJavaOptions", "-XX:+UseG1GC")
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .config("hive.exec.dynamic.partition", "true")
  .config("hive.exec.dynamic.partition.mode", "nonstrict")
  .enableHiveSupport()
  .getOrCreate();

spark.sql("set hive.exec.parallel=true")

// Create a Properties() object to hold the parameters.
val connectionProperties = new Properties()
connectionProperties.setProperty("Driver", driverClass)
connectionProperties.setProperty("fetchSize", "100000")

// read data from JDBC server and construct a dataframe
val jdbcDF1 = spark.read.jdbc(url = jdbcUrl, table = (select * from jdbcTable) e, properties = connectionProperties)

val jdbcDF = jdbcDF1.repartition(1000)

val count = jdbcDF.count()

println("red "+count+" records from sql server and started loading into hive")

// if count > 0 then insert the records into Hive
if (count > 0) {
  // create spark temporary table
  jdbcDF.createOrReplaceTempView("sparkTempTable")
  // insert into Hive external table
  spark.sql("insert into externalTable partition (hivePartitionCol) select * from sparkTempTable  distribute by  hivePartitionCol ")
}
println("completed the job for loading the data into hive")

spark.stop()

2 个答案:

答案 0 :(得分：2)

当随机数据块的大小超过2 GB的限制（火花无法处理）时，发生以下错误。

    Caused by: java.lang.IllegalArgumentException: Too large frame: 5211883372140375593
            at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119)
            at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)

选项1（> = Spark2.2）：

    与其他较低版本的Spark相比，

  1. >= Spark2.2以更好的方式处理了此问题。有关信息，请参见SPARK-19659。

  2. 使用以下Spark配置： 将spark.sql.shuffle.partitions的值从默认的200修改为大于2001的值。 将spark.default.parallelism的值设置为与spark.sql.shuffle.partitions相同的值。

选项2：

  1. 标识引起问题的DataFrame。
    一种。创建新的DataFrame后添加一个Spark操作（例如df.count()）。
    b。打印任何内容以检查DataFrame。
    C。如果未对DataFrame执行print语句，则问题出在该DataFrame上。

  2. 识别出DataFrame之后，使用df.repartition()重新分区DataFrame，然后使用df.cache()对其进行缓存。

  3. 如果数据存在偏斜，并且您使用的Spark版本早于2.2，请修改代码。

答案 1 :(得分：0)

发生此问题是因为有太大的物体无法随机播放。

您可以尝试增加Slack分区吗？

.config("spark.sql.shuffle.partitions", "1000")

或者您可以尝试添加此配置吗？

.config("spark.shuffle.spill.compress", true)

.config("spark.shuffle.compress", true)

或者您可以降低块大小以降低随机存储器的使用率

相关问题
最新问题