使用apache-spark的graphX API,我编写了以下代码,成功生成了Graph。但是,当我尝试查询此图时,会发生一些与内存相关的错误
15/10/09 12:10:55 INFO TaskSetManager: Starting task 6.0 in stage 38.0 (TID 90, localhost, PROCESS_LOCAL, 3940 bytes)
15/10/09 12:10:55 INFO Executor: Running task 6.0 in stage 38.0 (TID 90)
15/10/09 12:10:55 INFO TaskSetManager: Finished task 5.0 in stage 38.0 (TID 89) in 45114 ms on localhost (5/12)
15/10/09 12:10:55 INFO ShuffleBlockFetcherIterator: Getting 6 non-empty blocks out of 6 blocks
15/10/09 12:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/10/09 12:10:55 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
15/10/09 12:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/10/09 12:10:55 INFO ShuffleBlockFetcherIterator: Getting 6 non-empty blocks out of 6 blocks
15/10/09 12:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/10/09 12:11:06 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
15/10/09 12:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
15/10/09 12:16:59 INFO Executor: Finished task 6.0 in stage 38.0 (TID 90). 1011 bytes result sent to driver
15/10/09 12:16:59 ERROR Executor: Exception in task 4.0 in stage 38.0 (TID 88)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$join$1$$anonfun$apply$3$$anonfun$apply$4.apply(PairRDDFunctions.scala:485)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$join$1$$anonfun$apply$3$$anonfun$apply$4.apply(PairRDDFunctions.scala:485)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at org.apache.spark.util.collection.CompactBuffer$$anon$1.foreach(CompactBuffer.scala:113)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at org.apache.spark.util.collection.CompactBuffer.foreach(CompactBuffer.scala:28)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at org.apache.spark.util.collection.CompactBuffer.map(CompactBuffer.scala:28)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$join$1$$anonfun$apply$3.apply(PairRDDFunctions.scala:485)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$join$1$$anonfun$apply$3.apply(PairRDDFunctions.scala:485)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at org.apache.spark.util.collection.CompactBuffer$$anon$1.foreach(CompactBuffer.scala:113)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at org.apache.spark.util.collection.CompactBuffer.foreach(CompactBuffer.scala:28)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at org.apache.spark.util.collection.CompactBuffer.flatMap(CompactBuffer.scala:28)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$join$1.apply(PairRDDFunctions.scala:485)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$join$1.apply(PairRDDFunctions.scala:484)
at org.apache.spark.rdd.FlatMappedValuesRDD$$anonfun$compute$1.apply(FlatMappedValuesRDD.scala:32)
at org.apache.spark.rdd.FlatMappedValuesRDD$$anonfun$compute$1.apply(FlatMappedValuesRDD.scala:31)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:202)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
如果我尝试使用 graph.numvertices
,我会收到错误消息$scope.categories= [{"category_id":"1","category_name":"sports"},{"category_id":"2","category_name":"casual"},{"category_id"
:"3","category_name":"formal"},{"category_id":"4","category_name":"party wear"},{"category_id":"5","category_name"
:"winter"},{"category_id":"9","category_name":"summer"}]
请建议我在哪里犯了错误。
由于