<span title="Telephone number including area code."><td class="Data">Phone:</td></span>
我注意到内存中还剩49.7GB。为什么无法将14.8GB大小的广播变量存储到内存中？
存储在磁盘后，会出现异常
16/03/24 11:06:17 INFO TaskSetManager: Finished task 35.0 in stage 19.0 (TID 430) in 38746 ms on localhost (36/36)
16/03/24 11:06:17 INFO TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool
16/03/24 11:06:17 INFO DAGScheduler: ResultStage 19 (collect at hello.scala:599) finished in 411.119 s
16/03/24 11:06:17 INFO DAGScheduler: Job 7 finished: collect at hello.scala:599, took 1091.829070 s
16/03/24 11:08:40 INFO BlockManagerInfo: Removed broadcast_12_piece0 on localhost:51994 in memory (size: 1678.0 B, free: 49.7 GB)
16/03/24 11:08:40 INFO ContextCleaner: Cleaned accumulator 8
16/03/24 11:08:40 INFO BlockManagerInfo: Removed broadcast_19_piece0 on localhost:51994 in memory (size: 1567.0 B, free: 49.7 GB)
16/03/24 11:08:40 INFO ContextCleaner: Cleaned accumulator 12
16/03/24 11:08:40 INFO BlockManagerInfo: Removed broadcast_18_piece0 on localhost:51994 in memory (size: 2.3 KB, free: 49.7 GB)
16/03/24 11:08:40 INFO ContextCleaner: Cleaned accumulator 11
16/03/24 11:08:40 INFO ContextCleaner: Cleaned shuffle 3
16/03/24 11:08:40 INFO BlockManagerInfo: Removed broadcast_16_piece0 on localhost:51994 in memory (size: 1567.0 B, free: 49.7 GB)
16/03/24 11:08:40 INFO ContextCleaner: Cleaned accumulator 10
16/03/24 11:08:40 INFO BlockManagerInfo: Removed broadcast_15_piece0 on localhost:51994 in memory (size: 2.2 KB, free: 49.7 GB)
16/03/24 11:08:40 INFO ContextCleaner: Cleaned accumulator 9
16/03/24 11:08:40 INFO ContextCleaner: Cleaned shuffle 2
16/03/24 11:10:05 INFO MemoryStore: ensureFreeSpace(11112254799) called with curMem=15883105010, maxMem=55566516879
16/03/24 11:10:05 WARN MemoryStore: Not enough space to cache broadcast_20 in memory! (computed 31.1 GB so far)
16/03/24 11:10:05 INFO MemoryStore: Memory use = 14.8 GB (blocks) + 1024.0 KB (scratch space shared across 1 tasks(s)) = 14.8 GB. Storage limit = 51.8 GB.
16/03/24 11:10:05 WARN MemoryStore: Persisting block broadcast_20 to disk instead.
我确定数组长度小于16/03/24 11:10:05 WARN MemoryStore: Persisting block broadcast_20 to disk instead.
16/03/24 11:24:08 WARN BlockManager: Putting block broadcast_20 failed
Exception in thread "main" java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:828)
at org.apache.spark.storage.DiskStore$$anonfun$getBytes$2.apply(DiskStore.scala:125)
at org.apache.spark.storage.DiskStore$$anonfun$getBytes$2.apply(DiskStore.scala:113)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1206)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:127)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:134)
at org.apache.spark.storage.DiskStore.putIterator(DiskStore.scala:102)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:176)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:143)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:791)
at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:638)
at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:996)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:99)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:85)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
at hello$.main(hello.scala:599)
。