Question

在MacBook Pro（16gb mem）上使用kbastani/spark-neo4j和docker-compose，我试图分析图表的 strong_connected_components 。

我有一个包含大约60,000个节点(n1:Node {id:1})-[r:NEXT {count:100}]->(n2:Node {id:2})的图表。

使用neo4j浏览器，我设法将 pagerank 处理回我的节点。

但是，当我尝试运行更复杂的算法（如 strong_connected_components ）时，出现以下错误：

mazerunner_1  | 16/11/29 14:58:01 ERROR Utils: Uncaught exception in thread SparkListenerBus
mazerunner_1  | java.lang.OutOfMemoryError: Java heap space
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5$$anonfun$apply$9.apply(JobProgressListener.scala:200)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5$$anonfun$apply$9.apply(JobProgressListener.scala:200)
mazerunner_1  |     at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
mazerunner_1  |     at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5.apply(JobProgressListener.scala:200)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5.apply(JobProgressListener.scala:198)
mazerunner_1  |     at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
mazerunner_1  |     at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener.onJobStart(JobProgressListener.scala:198)
mazerunner_1  |     at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:34)
mazerunner_1  |     at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
mazerunner_1  |     at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
mazerunner_1  |     at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
mazerunner_1  |     at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
mazerunner_1  | Exception in thread "SparkListenerBus" java.lang.OutOfMemoryError: Java heap space
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5$$anonfun$apply$9.apply(JobProgressListener.scala:200)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5$$anonfun$apply$9.apply(JobProgressListener.scala:200)
mazerunner_1  |     at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
mazerunner_1  |     at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5.apply(JobProgressListener.scala:200)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener$$anonfun$onJobStart$5.apply(JobProgressListener.scala:198)
mazerunner_1  |     at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
mazerunner_1  |     at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
mazerunner_1  |     at org.apache.spark.ui.jobs.JobProgressListener.onJobStart(JobProgressListener.scala:198)
mazerunner_1  |     at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:34)
mazerunner_1  |     at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
mazerunner_1  |     at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
mazerunner_1  |     at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
mazerunner_1  |     at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618)
mazerunner_1  |     at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)

我试图修改我的docker-compose.yml文件，如下所示：

hdfs:
  environment:
    - "JAVA_OPTS=-Xmx5g"
  image: sequenceiq/hadoop-docker:2.4.1
  command: /etc/bootstrap.sh -d -bash
mazerunner:
  environment:
    - "JAVA_OPTS=-Xmx5g"
  image: kbastani/neo4j-graph-analytics:latest
  links:
   - hdfs
graphdb:
  environment:
    - "JAVA_OPTS=-Xmx2g"
  image: kbastani/docker-neo4j:latest
  ports:
   - "7474:7474"
   - "1337:1337"
  volumes:
   - /opt/data
  links:
   - mazerunner
   - hdfs

没有成功。我该如何配置spark＆amp; hdfs使用最大可用内存？

Answer 1

我的解决方案是增加虚拟机的内存大小。在我的Virtual Box UI上，我调整了“基本内存”大小滑块。

neo4j-mazerunner，如何在docker-compose.yml中增加内存大小

1 个答案: