Spark GraphX-PageRank算法：内存不足错误

Question

我是Spark Hadoop的初学者。我正在尝试使用PageRank算法进行1次迭代，但是对大图有问题。

我尝试运行一个由7个节点组成的集群，每个节点的NodeManager上的内存量为48G。

我的图有大约8000万个顶点和430百万个边。

我以...运行火花工作

./bin/spark-shell --executor-cores 5 --executor-memory 10G --num-executors 30

这是我的代码：

val edgesRDD: RDD[Edge[Int]] = edges.rdd.map { edge =>
  Edge(edge.get(0).toString.toLong, edge.get(1).toString.toLong, 0)
}
val verticesRDD: RDD[(VertexId, Int)] = vertices.rdd.map { vertex =>
  (vertex.get(0).toString.toLong, 0)
}
val graph = Graph(verticesRDD, edgesRDD)
val resPR = graph.staticPageRank(1)

以下是异常详细信息：

# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill %p"
#   Executing /bin/sh -c "kill 56758"...
2018-09-05 17:29:57 WARN  TransportChannelHandler:78 - Exception in connection from IDC-vbidatanode44/10.60.170.44:40934
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at io.netty.buffer.CompositeByteBuf.nioBuffer(CompositeByteBuf.java:1466)
    at io.netty.buffer.AbstractDerivedByteBuf.nioBuffer(AbstractDerivedByteBuf.java:113)
    at io.netty.buffer.AbstractByteBuf.nioBuffer(AbstractByteBuf.java:1203)
    at org.apache.spark.network.buffer.NettyManagedBuffer.nioByteBuffer(NettyManagedBuffer.java:45)
    at org.apache.spark.network.BlockTransferService$$anon$1.onBlockFetchSuccess(BlockTransferService.scala:109)
    at org.apache.spark.network.shuffle.RetryingBlockFetcher$RetryingBlockFetchListener.onBlockFetchSuccess(RetryingBlockFetcher.java:204)
    at org.apache.spark.network.shuffle.OneForOneBlockFetcher$ChunkCallback.onSuccess(OneForOneBlockFetcher.java:97)
    at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:171)
    at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:120)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
2018-09-05 17:29:57 ERROR CoarseGrainedExecutorBackend:43 - RECEIVED SIGNAL TERM
2018-09-05 17:29:57 INFO  DiskBlockManager:54 - Shutdown hook called
2018-09-05 17:29:57 INFO  ShutdownHookManager:54 - Shutdown hook called

有什么主意我该如何停止收到此错误？

谢谢。