我已经下载了spark-2.4.0.tgz
,它无法通过构建的spark-sql
模块。
对我的构建环境进行了以下优化:
-DskipTests
一起跑Top
报告运行时3.8G物理内存可用 为进程显式分配大量内存
`MAVEN_OPTS="-Xmx3500m" mvn -DskipTests package`
Activity Monitor
表明它仅在使用1.5GB
:因此,从system
的角度以及从它自己分配的内存来看,该进程都有许多未使用的可用内存 li>
其他详细信息:
High Sierra
Memory
和Swap
总共仅使用11GB 这是堆栈跟踪:
[warn] /shared/spark/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala:315: method apply in object ProcessingTime is deprecated: use Trigger.ProcessingTime(interval)
[warn] StartStream(ProcessingTime("10 seconds"), new StreamManualClock),
[warn] ^
[warn] /shared/spark/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala:356: method apply in object ProcessingTime is deprecated: use Trigger.ProcessingTime(interval)
[warn] StartStream(ProcessingTime("10 seconds"), new StreamManualClock(60 * 1000)),
[warn] ^
java.lang.OutOfMemoryError: GC overhead limit exceeded
at scala.tools.nsc.backend.icode.BasicBlocks$BasicBlock.<init>(BasicBlocks.scala:464)
at scala.tools.nsc.backend.icode.Members$Code.newBlock(Members.scala:97)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase$Context.newBlock(GenICode.scala:1907)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.genLoadIf(GenICode.scala:354)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genLoad(GenICode.scala:536)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genStat(GenICode.scala:181)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genStat$1.apply(GenICode.scala:155)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genStat$1.apply(GenICode.scala:155)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genLoad(GenICode.scala:922)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:123)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:71)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:148)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:98)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:71)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:89)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:67)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.apply(GenICode.scala:63)
at scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:467)
at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:458)
at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:467)
at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:425)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.run(GenICode.scala:55)
at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1528)
at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1513)
at scala.tools.nsc.Global$Run.compileSources(Global.scala:1508)
at scala.tools.nsc.Global$Run.compile(Global.scala:1609)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 2.4.0:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.569 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 1.136 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 0.499 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 0.658 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 1.903 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 0.503 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 0.590 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 2.484 s]
[INFO] Spark Project Core ................................. SUCCESS [ 16.434 s]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 0.651 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 1.977 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 4.197 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 17.791 s]
[INFO] Spark Project SQL .................................. FAILURE [03:49 min]
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:41 min
[INFO] Finished at: 2018-12-04T04:37:20-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed.: CompileFailed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-sql_2.11