Question

我试图在Windows 7上构建Spark 1.1.0并且运行sbt assembly失败并出现以下java.lang.OutOfMemoryError: Java heap space错误：

...    
[warn] Merging 'parquet\schema\Type$Repetition$3.class' with strategy 'first'
[warn] Merging 'parquet\schema\Type$Repetition.class' with strategy 'first'
[warn] Merging 'parquet\schema\Type.class' with strategy 'first'
[warn] Merging 'parquet\schema\TypeConverter.class' with strategy 'first'
[warn] Merging 'parquet\schema\TypeVisitor.class' with strategy 'first'
[warn] Merging 'plugin.properties' with strategy 'first'
[warn] Merging 'plugin.xml' with strategy 'first'
[warn] Merging 'reference.conf' with strategy 'concat'
[warn] Merging 'rootdoc.txt' with strategy 'first'
[warn] Strategy 'concat' was applied to a file
[warn] Strategy 'discard' was applied to 1723 files
[warn] Strategy 'first' was applied to 2713 files
[info] Assembly up to date: D:\app\spark\assembly\target\scala-2.10\spark-assembly-1.1.1-SNAPSHOT-hadoop1.0.4.jar
[info] Packaging D:\app\spark\examples\target\scala-2.10\spark-examples-1.1.1-SNAPSHOT-hadoop1.0.4.jar ...
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(Unknown Source)
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:196)
        at sbt.Execute.next$1(Execute.scala:88)
        ...
        at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
        at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
        at sbt.Aggregation$$anonfun$evaluatingParser$4$$anonfun$apply$5.apply(Ag
        ...
        at xsbt.boot.Boot$.main(Boot.scala:17)
        at xsbt.boot.Boot.main(Boot.scala)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:197)
        at scala.collection.TraversableOnce$$anonfun$addString$1.apply(TraversableOnce.scala:327)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:320)
        at scala.collection.AbstractTraversable.addString(Traversable.scala:105)
        at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:286)
        at scala.collection.AbstractTraversable.mkString(Traversable.scala:105)
        at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:288)
        at scala.collection.AbstractTraversable.mkString(Traversable.scala:105)
        at sbt.Package$.sourcesDebugString(Package.scala:111)
        at sbt.Package$$anonfun$makeJar$2.apply(Package.scala:106)
        at sbt.Package$$anonfun$makeJar$2.apply(Package.scala:106)
        at sbt.MultiLogger.log(MultiLogger.scala:31)
        at sbt.Logger$class.debug(Logger.scala:121)
        at sbt.AbstractLogger.debug(Logger.scala:11)
        at sbt.Package$.makeJar(Package.scala:106)
        at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$makeJar$1(Plugin.scala:187)
        at sbtassembly.Plugin$Assembly$$anonfun$7$$anonfun$apply$6.apply(Plugin.scala:226)
        at sbtassembly.Plugin$Assembly$$anonfun$7$$anonfun$apply$6.apply(Plugin.scala:222)
        at sbt.Tracked$$anonfun$outputChanged$1.apply(Tracked.scala:57)
        at sbt.Tracked$$anonfun$outputChanged$1.apply(Tracked.scala:52)
        at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
        at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
        at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
[error] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Jav
a heap space
[error] Use 'last' for the full log.

我尝试将-Xmx1024m添加到sbt.bat文件，但它没有解决问题。

Answer 1

从日志中可以看出您正在尝试从来源构建1.1.1-SNAPSHOT版本。

[info]最新组装：D：\ app \ spark \ assembly \ target \ scala-2.10 \ spark-assembly-1.1.1-SNAPSHOT-hadoop1.0.4.jar

如果您想使用 Spark我强烈建议您从http://spark.apache.org/downloads.html下载适用于您操作系统的软件包。

如果你坚持自己从源代码构建Spark，你应该考虑将构建工具从sbt更改为Apache Maven。鉴于GitHub上的Building Spark和Spark文档中的Building Spark with Maven，他们似乎已经切换到Apache Maven作为最终用户的构建工具。

使用Maven构建Spark并遵循指南：

您需要将Maven配置为使用比平常更多的内存设置MAVEN_OPTS。我们建议使用以下设置：
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

它应该也适用于sbt。使用SBT_OPTS并重新开始。

如何在Windows 7上构建Spark 1.1.0？

1 个答案: