由于OOME无法从源代码构建Spark

时间:2018-12-04 12:44:47

标签: apache-spark

我已经下载了spark-2.4.0.tgz,它无法通过构建的spark-sql模块。

对我的构建环境进行了以下优化:

  • 运行锌
  • -DskipTests一起跑
  • 淘汰了主要的记忆猪计划
    • Top报告运行时3.8G物理内存可用
  • 为进程显式分配大量内存

    `MAVEN_OPTS="-Xmx3500m" mvn -DskipTests package`
    
  • 但是Activity Monitor表明它仅在使用1.5GB:因此,从system的角度以及从它自己分配的内存来看,该进程都有许多未使用的可用内存 li>

其他详细信息:

  • 在Macos High Sierra
  • 16GB物理RAM
  • 我通常运行的程序比当时运行的程序多10GB(MemorySwap总共仅使用11GB

这是堆栈跟踪:

[warn] /shared/spark/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala:315: method apply in object ProcessingTime is deprecated: use Trigger.ProcessingTime(interval)
[warn]       StartStream(ProcessingTime("10 seconds"), new StreamManualClock),
[warn]                   ^
[warn] /shared/spark/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala:356: method apply in object ProcessingTime is deprecated: use Trigger.ProcessingTime(interval)
[warn]       StartStream(ProcessingTime("10 seconds"), new StreamManualClock(60 * 1000)),
[warn]                   ^
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at scala.tools.nsc.backend.icode.BasicBlocks$BasicBlock.<init>(BasicBlocks.scala:464)
    at scala.tools.nsc.backend.icode.Members$Code.newBlock(Members.scala:97)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase$Context.newBlock(GenICode.scala:1907)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.genLoadIf(GenICode.scala:354)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genLoad(GenICode.scala:536)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genStat(GenICode.scala:181)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genStat$1.apply(GenICode.scala:155)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genStat$1.apply(GenICode.scala:155)
    at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
    at scala.collection.immutable.List.foldLeft(List.scala:84)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genLoad(GenICode.scala:922)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:123)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:71)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:148)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:98)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:71)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:89)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:67)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.apply(GenICode.scala:63)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:467)
    at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:458)
    at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:467)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:425)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.run(GenICode.scala:55)
    at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1528)
    at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1513)
    at scala.tools.nsc.Global$Run.compileSources(Global.scala:1508)
    at scala.tools.nsc.Global$Run.compile(Global.scala:1609)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 2.4.0:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.569 s]
[INFO] Spark Project Tags ................................. SUCCESS [  1.136 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  0.499 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  0.658 s]
[INFO] Spark Project Networking ........................... SUCCESS [  1.903 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  0.503 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  0.590 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  2.484 s]
[INFO] Spark Project Core ................................. SUCCESS [ 16.434 s]
[INFO] Spark Project ML Local Library ..................... SUCCESS [  0.651 s]
[INFO] Spark Project GraphX ............................... SUCCESS [  1.977 s]
[INFO] Spark Project Streaming ............................ SUCCESS [  4.197 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 17.791 s]
[INFO] Spark Project SQL .................................. FAILURE [03:49 min]
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:41 min
[INFO] Finished at: 2018-12-04T04:37:20-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed.: CompileFailed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-sql_2.11

0 个答案:

没有答案