为什么群集模式比Spark中的客户端模式慢得多？

Question

我有使用GraphFrame创建的15k图形对象。这些图非常小（每个图10-100个节点，10-100个边）。我只需要在每个图上应用最短路径和某些其他算法。我目前的实现是使用for循环遍历所有15k图。令人惊讶的是，群集模式比客户端模式慢得多。

我处理了10张图表。以下是客户端模式和群集模式的性能指标。我试图找到为什么群集模式比客户端模式慢。我只是无法理解指标。

群集配置（Spark Standalone）

客户端模式的配置：

客户端模式：

Scheduling mode = FIFO
Spark Context default degree of parallelism = 4
Aggregated Spark stage metrics:
numStages => 1019
sum(numTasks) => 3263
elapsedTime => 38675 (39 s)
sum(stageDuration) => 19177 (19 s)
sum(executorRunTime) => 41413 (41 s)
sum(executorCpuTime) => 6305 (6 s)
sum(executorDeserializeTime) => 6386 (6 s)
sum(executorDeserializeCpuTime) => 3382 (3 s)
sum(resultSerializationTime) => 82 (82 ms)
sum(jvmGCTime) => 1171 (1 s)
sum(shuffleFetchWaitTime) => 3 (3 ms)
sum(shuffleWriteTime) => 603 (0.6 s)
max(resultSize) => 23947 (23.0 KB)
sum(numUpdatedBlockStatuses) => 1848
sum(diskBytesSpilled) => 0 (0 Bytes)
sum(memoryBytesSpilled) => 0 (0 Bytes)
max(peakExecutionMemory) => 35651584
sum(recordsRead) => 16972
sum(bytesRead) => 22064469 (21.0 MB)
sum(recordsWritten) => 0
sum(bytesWritten) => 0 (0 Bytes)
sum(shuffleTotalBytesRead) => 1431309 (1397.0 KB)
sum(shuffleTotalBlocksFetched) => 4304
sum(shuffleLocalBlocksFetched) => 4304
sum(shuffleRemoteBlocksFetched) => 0
sum(shuffleBytesWritten) => 1205067 (1176.0 KB)
sum(shuffleRecordsWritten) => 8228

群集模式

Scheduling mode = FIFO
Spark Context default degree of parallelism = 16
Aggregated Spark stage metrics:
numStages => 1019
sum(numTasks) => 3371
elapsedTime => 119895 (2.0 min)
sum(stageDuration) => 114501 (1.9 min)
sum(executorRunTime) => 162486 (2.7 min)
sum(executorCpuTime) => 29376 (29 s)
sum(executorDeserializeTime) => 74342 (1.2 min)
sum(executorDeserializeCpuTime) => 11919 (12 s)
sum(resultSerializationTime) => 87 (87 ms)
sum(jvmGCTime) => 7422 (7 s)
sum(shuffleFetchWaitTime) => 20899 (21 s)
sum(shuffleWriteTime) => 1144 (1 s)
max(resultSize) => 61683 (60.0 KB)
sum(numUpdatedBlockStatuses) => 6097
sum(diskBytesSpilled) => 0 (0 Bytes)
sum(memoryBytesSpilled) => 0 (0 Bytes)
max(peakExecutionMemory) => 47185920
sum(recordsRead) => 16962
sum(bytesRead) => 22037962 (21.0 MB)
sum(recordsWritten) => 0
sum(bytesWritten) => 0 (0 Bytes)
sum(shuffleTotalBytesRead) => 1480503 (1445.0 KB)
sum(shuffleTotalBlocksFetched) => 4571
sum(shuffleLocalBlocksFetched) => 2361
sum(shuffleRemoteBlocksFetched) => 2210
sum(shuffleBytesWritten) => 1211392 (1183.0 KB)
sum(shuffleRecordsWritten) => 8228

Answer 1

TL; DR 您正在使用错误的工具。

非常小的图形（ 10-100个节点，每个图形的10-100个边缘）根本无法从分布式处理中受益。

与此同时，像Spark这样的分布式系统中使用的高成本抽象增加了大量开销，使其成为主导成本。

一般来说：

• 分布式图处理很难。
• 在大多数情况下，扩展胜出超过扩展。