JVM OutOfMemory - 为Chunk :: new请求了8589934608个字节。交换空间？

Question

在我的公司，我们在Spring（Flex前端）开发企业Web应用程序，并以SAAS风格将此应用程序部署到Tomcat 7中的客户。

最近我们在stderr日志文件中看到（看似）随机OutOfMemory错误：

java.lang.OutOfMemoryError

这个问题发生在内存的不同配置上，我们看到此错误时配置的内存如下：

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 8589934608 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (allocation.cpp:215), pid=1184, tid=1692
#  Error: Chunk::new
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode windows-amd64 )
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000005864800):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=1692, stack(0x0000000005d00000,0x0000000005e00000)]

Stack: [0x0000000005d00000,0x0000000005e00000]

Current CompileTask:
C2:6164  !   org.apache.catalina.servlets.DefaultServlet.serveResource(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V (1261 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x0000000006569000 JavaThread "http-bio-8443-exec-1007" daemon [_thread_in_native, id=2876, stack(0x000000000f800000,0x000000000f900000)]
  0x000000000656a800 JavaThread "http-bio-8443-exec-1006" daemon [_thread_in_native, id=4012, stack(0x000000000ef80000,0x000000000f080000)]
  0x000000000656a000 JavaThread "http-bio-8443-exec-1005" daemon [_thread_in_native, id=4068, stack(0x000000000ec00000,0x000000000ed00000)]
  0x000000000656c000 JavaThread "http-bio-8443-exec-1004" daemon [_thread_in_native, id=2852, stack(0x000000000eb00000,0x000000000ec00000)]
  0x0000000008a29800 JavaThread "http-bio-8443-exec-1003" daemon [_thread_in_native, id=1796, stack(0x000000000ea00000,0x000000000eb00000)]
  0x0000000008a2a000 JavaThread "http-bio-8443-exec-1002" daemon [_thread_in_native, id=792, stack(0x000000000e800000,0x000000000e900000)]
  0x0000000008a28000 JavaThread "http-bio-8443-exec-1001" daemon [_thread_blocked, id=3624, stack(0x000000000e600000,0x000000000e700000)]
  0x0000000008a27000 JavaThread "http-bio-8443-exec-1000" daemon [_thread_blocked, id=468, stack(0x000000000e500000,0x000000000e600000)]
  0x0000000008a2d000 JavaThread "http-bio-8443-exec-999" daemon [_thread_blocked, id=1436, stack(0x000000000cb60000,0x000000000cc60000)]
  0x0000000006567000 JavaThread "http-bio-8443-exec-998" daemon [_thread_in_Java, id=3520, stack(0x00000000084d0000,0x00000000085d0000)]
  0x0000000006bb0800 JavaThread "http-bio-8443-exec-993" daemon [_thread_in_native, id=1848, stack(0x000000000f500000,0x000000000f600000)]
  0x0000000006bb1800 JavaThread "http-bio-8443-exec-992" daemon [_thread_blocked, id=2944, stack(0x000000000f300000,0x000000000f400000)]
  0x0000000006bb4800 JavaThread "http-bio-8443-exec-989" daemon [_thread_in_native, id=3092, stack(0x000000000cc60000,0x000000000cd60000)]
  0x0000000006bb2800 JavaThread "http-bio-8443-exec-987" daemon [_thread_blocked, id=3676, stack(0x000000000f700000,0x000000000f800000)]
  0x0000000008a2c000 JavaThread "http-bio-8443-exec-981" daemon [_thread_in_native, id=668, stack(0x000000000e700000,0x000000000e800000)]
  0x0000000006568800 JavaThread "http-bio-8443-exec-971" daemon [_thread_in_native, id=3664, stack(0x0000000000b30000,0x0000000000c30000)]
  0x000000000b0e6800 JavaThread "http-bio-8443-exec-928" daemon [_thread_in_native, id=3760, stack(0x000000000f600000,0x000000000f700000)]
  0x0000000006baf000 JavaThread "http-bio-8443-exec-892" daemon [_thread_in_native, id=4080, stack(0x000000000e900000,0x000000000ea00000)]
  0x0000000008a26800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=624, stack(0x000000000f400000,0x000000000f500000)]
  0x0000000008a2a800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=1476, stack(0x000000000f100000,0x000000000f200000)]
  0x0000000008a28800 JavaThread "scheduler-2" [_thread_blocked, id=3036, stack(0x00000000083d0000,0x00000000084d0000)]
  0x000000000b0e0800 JavaThread "ajp-bio-8009-AsyncTimeout" daemon [_thread_blocked, id=2588, stack(0x000000000e400000,0x000000000e500000)]
  0x000000000b0e0000 JavaThread "ajp-bio-8009-Acceptor-0" daemon [_thread_in_native, id=2584, stack(0x000000000d260000,0x000000000d360000)]
  0x000000000b0df800 JavaThread "http-bio-8443-AsyncTimeout" daemon [_thread_blocked, id=2580, stack(0x000000000d160000,0x000000000d260000)]
  0x0000000008fdf800 JavaThread "http-bio-8443-Acceptor-0" daemon [_thread_in_native, id=2576, stack(0x000000000d060000,0x000000000d160000)]
  0x0000000008fdc800 JavaThread "http-bio-8080-AsyncTimeout" daemon [_thread_blocked, id=2572, stack(0x000000000cf60000,0x000000000d060000)]
  0x0000000008fde000 JavaThread "http-bio-8080-Acceptor-0" daemon [_thread_in_native, id=2568, stack(0x000000000ce60000,0x000000000cf60000)]
  0x0000000008fdd800 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=2564, stack(0x000000000cd60000,0x000000000ce60000)]
  0x0000000008fdf000 JavaThread "scheduler-1" [_thread_blocked, id=2536, stack(0x0000000000c30000,0x0000000000d30000)]
  0x0000000008fda800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#19" daemon [_thread_blocked, id=2476, stack(0x000000000ca60000,0x000000000cb60000)]
  0x0000000008fd9800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#18" daemon [_thread_blocked, id=2472, stack(0x000000000c960000,0x000000000ca60000)]
  0x0000000008fd9000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#17" daemon [_thread_blocked, id=2468, stack(0x000000000c860000,0x000000000c960000)]
  0x0000000008fd8800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#16" daemon [_thread_blocked, id=2464, stack(0x000000000c760000,0x000000000c860000)]
  0x0000000008fd7800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#15" daemon [_thread_blocked, id=2460, stack(0x000000000c660000,0x000000000c760000)]
  0x0000000008fd7000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#14" daemon [_thread_blocked, id=2456, stack(0x000000000c560000,0x000000000c660000)]
  0x0000000008fd6000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#13" daemon [_thread_blocked, id=2452, stack(0x000000000c460000,0x000000000c560000)]
  0x0000000008fd5800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#12" daemon [_thread_blocked, id=2448, stack(0x000000000c360000,0x000000000c460000)]
  0x0000000008fd4800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#11" daemon [_thread_blocked, id=2444, stack(0x000000000c260000,0x000000000c360000)]
  0x0000000008fd4000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#10" daemon [_thread_blocked, id=2440, stack(0x000000000c160000,0x000000000c260000)]
  0x0000000008fd3000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#9" daemon [_thread_blocked, id=2436, stack(0x000000000c060000,0x000000000c160000)]
  0x0000000008fd2800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#8" daemon [_thread_blocked, id=2432, stack(0x000000000bf60000,0x000000000c060000)]
  0x0000000008fd2000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#7" daemon [_thread_blocked, id=2428, stack(0x000000000be60000,0x000000000bf60000)]
  0x0000000008fd1000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#6" daemon [_thread_blocked, id=2424, stack(0x000000000bd60000,0x000000000be60000)]
  0x00000000059d4800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#5" daemon [_thread_blocked, id=2420, stack(0x000000000bc60000,0x000000000bd60000)]
  0x00000000059d3800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#4" daemon [_thread_blocked, id=2416, stack(0x000000000bb60000,0x000000000bc60000)]
  0x00000000066b1800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#3" daemon [_thread_blocked, id=2412, stack(0x0000000009aa0000,0x0000000009ba0000)]
  0x000000000682b000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#2" daemon [_thread_blocked, id=2408, stack(0x00000000099a0000,0x0000000009aa0000)]
  0x000000000668a000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#1" daemon [_thread_blocked, id=2404, stack(0x00000000098a0000,0x00000000099a0000)]
  0x000000000872b000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-HelperThread-#0" daemon [_thread_blocked, id=2400, stack(0x00000000097a0000,0x00000000098a0000)]
  0x0000000006727800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->2rxcpk971gv269anlzvh0|79a422d9]-AdminTaskTimer" daemon [_thread_blocked, id=2396, stack(0x00000000096a0000,0x00000000097a0000)]
  0x000000000629e800 JavaThread "Thread-2" daemon [_thread_blocked, id=2292, stack(0x00000000085d0000,0x00000000086d0000)]
  0x00000000059b4000 JavaThread "GC Daemon" daemon [_thread_blocked, id=2216, stack(0x0000000006c00000,0x0000000006d00000)]
  0x0000000005866800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1700, stack(0x0000000005e00000,0x0000000005f00000)]
=>0x0000000005864800 JavaThread "CompilerThread1" daemon [_thread_in_native, id=1692, stack(0x0000000005d00000,0x0000000005e00000)]
  0x0000000005859800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1680, stack(0x0000000005c00000,0x0000000005d00000)]
  0x0000000005856800 JavaThread "Attach Listener" daemon [_thread_blocked, id=1672, stack(0x0000000005b00000,0x0000000005c00000)]
  0x0000000005851800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1668, stack(0x0000000005a00000,0x0000000005b00000)]
  0x0000000000473000 JavaThread "Finalizer" daemon [_thread_blocked, id=1516, stack(0x0000000005700000,0x0000000005800000)]
  0x0000000000472000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1508, stack(0x0000000005600000,0x0000000005700000)]
  0x000000000037c800 JavaThread "main" [_thread_in_native, id=1376, stack(0x0000000000f30000,0x0000000001030000)]

Other Threads:
  0x000000000046d000 VMThread [stack: 0x0000000005500000,0x0000000005600000] [id=1492]
  0x0000000005867800 WatcherThread [stack: 0x0000000005f00000,0x0000000006000000] [id=1704]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen      total 674624K, used 270750K [0x0000000115550000, 0x000000013fff0000, 0x000000013fff0000)
  eden space 649152K, 40% used [0x0000000115550000,0x0000000125927478,0x000000013cf40000)
  from space 25472K, 18% used [0x000000013e710000,0x000000013eba0578,0x000000013fff0000)
  to   space 24384K, 0% used [0x000000013cf40000,0x000000013cf40000,0x000000013e710000)
 PSOldGen        total 1398144K, used 1102711K [0x00000000bfff0000, 0x0000000115550000, 0x0000000115550000)
  object space 1398144K, 78% used [0x00000000bfff0000,0x00000001034cdd10,0x0000000115550000)
 PSPermGen       total 1048576K, used 214588K [0x000000007fff0000, 0x00000000bfff0000, 0x00000000bfff0000)
  object space 1048576K, 20% used [0x000000007fff0000,0x000000008d17f088,0x00000000bfff0000)

Dynamic libraries:
0x0000000140000000 - 0x000000014001c000     C:\Program Files\Apache Software Foundation\Tomcat 7.0\bin\Tomcat7.exe
0x00000000770f0000 - 0x0000000077299000     C:\Windows\SYSTEM32\ntdll.dll
0x0000000076ed0000 - 0x0000000076fef000     C:\Windows\system32\kernel32.dll
0x000007fefd010000 - 0x000007fefd07c000     C:\Windows\system32\KERNELBASE.dll
0x000007fefd3b0000 - 0x000007fefd48b000     C:\Windows\system32\ADVAPI32.dll
0x000007fefeb20000 - 0x000007fefebbf000     C:\Windows\system32\msvcrt.dll
0x000007fefd910000 - 0x000007fefd92f000     C:\Windows\SYSTEM32\sechost.dll
0x000007fefd930000 - 0x000007fefda5d000     C:\Windows\system32\RPCRT4.dll
0x000007fefdc70000 - 0x000007fefe9f8000     C:\Windows\system32\SHELL32.dll
0x000007fefd890000 - 0x000007fefd901000     C:\Windows\system32\SHLWAPI.dll
0x000007fefd340000 - 0x000007fefd3a7000     C:\Windows\system32\GDI32.dll
0x0000000076ff0000 - 0x00000000770ea000     C:\Windows\system32\USER32.dll
0x000007fefee70000 - 0x000007fefee7e000     C:\Windows\system32\LPK.dll
0x000007fefd270000 - 0x000007fefd339000     C:\Windows\system32\USP10.dll
0x000007fefd810000 - 0x000007fefd83e000     C:\Windows\system32\IMM32.DLL
0x000007fefee90000 - 0x000007fefef99000     C:\Windows\system32\MSCTF.dll
0x000000006d7f0000 - 0x000000006de95000     C:\Program Files\Java\jre6\bin\server\jvm.dll
0x000007fef9900000 - 0x000007fef993b000     C:\Windows\system32\WINMM.dll
0x000000006d760000 - 0x000000006d76e000     C:\Program Files\Java\jre6\bin\verify.dll
0x000000006d3b0000 - 0x000000006d3d7000     C:\Program Files\Java\jre6\bin\java.dll
0x000000006d310000 - 0x000000006d31a000     C:\Program Files\Java\jre6\bin\hpi.dll
0x00000000772c0000 - 0x00000000772c7000     C:\Windows\system32\PSAPI.DLL
0x000000006d7b0000 - 0x000000006d7c2000     C:\Program Files\Java\jre6\bin\zip.dll
0x000007fefda60000 - 0x000007fefdc63000     C:\Windows\system32\ole32.dll
0x000000006d540000 - 0x000000006d54a000     C:\Program Files\Java\jre6\bin\management.dll
0x000000006d600000 - 0x000000006d617000     C:\Program Files\Java\jre6\bin\net.dll
0x000007fefd840000 - 0x000007fefd88d000     C:\Windows\system32\WS2_32.dll
0x000007fefee80000 - 0x000007fefee88000     C:\Windows\system32\NSI.dll
0x000007fefc650000 - 0x000007fefc6a5000     C:\Windows\system32\mswsock.dll
0x000007fefc640000 - 0x000007fefc647000     C:\Windows\System32\wship6.dll
0x000007fefc050000 - 0x000007fefc057000     C:\Windows\System32\wshtcpip.dll
0x000007fefc6b0000 - 0x000007fefc6c7000     C:\Windows\system32\CRYPTSP.dll
0x000007fefc3b0000 - 0x000007fefc3f7000     C:\Windows\system32\rsaenh.dll
0x000007fefceb0000 - 0x000007fefcebf000     C:\Windows\system32\profapi.dll
0x000007fefcd10000 - 0x000007fefcd1f000     C:\Windows\system32\CRYPTBASE.dll
0x000007fefbbb0000 - 0x000007fefbbc5000     C:\Windows\system32\NLAapi.dll
0x000007fef9940000 - 0x000007fef9955000     C:\Windows\system32\napinsp.dll
0x000007fefc4d0000 - 0x000007fefc52b000     C:\Windows\system32\DNSAPI.dll
0x000007fef9e40000 - 0x000007fef9e4b000     C:\Windows\System32\winrnr.dll
0x000007fefa4a0000 - 0x000007fefa4c7000     C:\Windows\system32\IPHLPAPI.DLL
0x000007fefa3f0000 - 0x000007fefa3fb000     C:\Windows\system32\WINNSI.DLL
0x000007fef9e30000 - 0x000007fef9e38000     C:\Windows\system32\rasadhlp.dll
0x000007fefa380000 - 0x000007fefa3d3000     C:\Windows\System32\fwpuclnt.dll
0x000000006d000000 - 0x000000006d1c2000     C:\Program Files\Java\jre6\bin\awt.dll
0x000007fef84c0000 - 0x000007fef8531000     C:\Windows\system32\WINSPOOL.DRV
0x000007fefb3a0000 - 0x000007fefb440000     C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7601.18201_none_a4d3b9377117c3df\COMCTL32.dll
0x000000006d2a0000 - 0x000000006d30b000     C:\Program Files\Java\jre6\bin\fontmanager.dll
0x000000006d620000 - 0x000000006d62b000     C:\Program Files\Java\jre6\bin\nio.dll
0x000007fefae80000 - 0x000007fefae98000     C:\Windows\system32\DWMAPI.DLL

VM Arguments:
jvm_args: -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 7.0 -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 7.0 -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 7.0\endorsed -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 7.0\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 7.0\conf\logging.properties -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 7.0 -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 7.0 -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 7.0\endorsed -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 7.0\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 7.0\conf\logging.properties -XX:PermSize=1024m -Xms2048m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError 
java_command: <unknown>
Launcher Type: generic

Environment Variables:
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 44 Stepping 2, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows Server 2008 R2 Build 7601 Service Pack 1

CPU:total 4 (1 cores per cpu, 1 threads per core) family 6 model 44 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt

Memory: 4k page, physical 8388088k(197084k free), swap 22991520k(8404948k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (16.0-b13) for windows-amd64 JRE (1.6.0_18-b07), built on Dec 17 2009 13:24:11 by "java_re" with MS VC++ 8.0 (VS2005)

time: Thu Feb 26 17:11:51 2015
elapsed time: 136555 seconds

这看起来不像是“正常”的OOM错误，因为它不会触发堆转储的保存，看起来更像是由OS触发的。 它看起来很奇怪JVM它正在请求SO MUCH内存，它是8,5 GB的内存，这类似于但是优于OS中配置的当前页面文件大小（8191 mb）

JVM选项如下：

  

-XX：PermSize =1024米   -Xms2048m   -Xmx2048m   -XX：+ HeapDumpOnOutOfMemoryError

堆和permgen空间对我来说并不完整，它引起我注意的是错误日志中的系统信息：

Memory: 4k page, physical 8388088k(197084k free), swap 22991520k(8404948k free)

显示了免费交换空间实际上小于请求的空间。

但为什么要求这么多？

JVM是java 1.6.18

当服务器受压或几乎空闲时，错误会随机出现。