我们在服务器崩溃时遇到以下错误
我发现一个重要的事情是，在每次垃圾收集之后，分配失败值会增加，并且在某些时候jvm会崩溃。 关于这个的任何想法
101402.988: [GC (Allocation Failure) 101402.988: [DefNew: 207886K->12434K(219456K), 0.0330469 secs]101403.021: [Tenured: 488730K->418985K(488808K), 0.8195576 secs] 694094K->418985K(708264K), [Metaspace: 247627K->247627K(1263616K)]OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f32aa000, 214532096, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 214532096 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/tomcat/logs/hs_err_pid16209.log
hs错误文件始终是相同的错误 我们已经尝试过增加内存，并且还在探索解决此问题的各种方案 但仍然存在问题
There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 214532096 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2638), pid=16209, tid=140213871417088
#
# JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007f86100a4000): VMThread [stack: 0x00007f8615ef9000,0x00007f8615ffa000] [id=16211]
Stack: [0x00007f8615ef9000,0x00007f8615ffa000], sp=0x00007f8615ff8510, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xa0854d]
V [libjvm.so+0x4c4cca]
V [libjvm.so+0x86c2d0]
V [libjvm.so+0x8669ce]
V [libjvm.so+0xa05344]
V [libjvm.so+0x5af7ed]
V [libjvm.so+0x5afb6d]
V [libjvm.so+0x5a01e0]
V [libjvm.so+0x4570a2]
V [libjvm.so+0xa094f1]
V [libjvm.so+0xa0e707]
V [libjvm.so+0xa0d0f8]
V [libjvm.so+0xa0d60a]
V [libjvm.so+0xa0da21]
V [libjvm.so+0x869702]
VM_Operation (0x00007f85ed7ace50): GenCollectForAllocation, mode: safepoint, requested by thread 0x00007f85d14e4000
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f85d04b5800 JavaThread "http-bio-8080-exec-699" daemon [_thread_blocked, id=30034, stack(0x00007f85cb0eb000,0x00007f85cb1ec000)]
0x00007f85daaa8800 JavaThread "http-bio-8080-exec-700" daemon [_thread_blocked, id=30033, stack(0x00007f85cbee1000,0x00007f85cbfe2000)]
0x00007f85d19e4000 JavaThread "http-bio-8080-exec-701" daemon [_thread_blocked, id=30032, stack(0x00007f85cc0e3000,0x00007f85cc1e4000)]
0x00007f85d1ab3800 JavaThread "http-bio-8080-exec-702" daemon [_thread_blocked, id=30031, stack(0x00007f85cc7ea000,0x00007f85cc8eb000)]
0x00007f85db4c5000 JavaThread "http-bio-8080-exec-714" daemon [_thread_blocked, id=30030, stack(0x00007f85cc8eb000,0x00007f85cc9ec000)]
0x00007f85d19e5800 JavaThread "http-bio-8080-exec-713" daemon [_thread_blocked, id=30029, stack(0x00007f85ccddc000,0x00007f85ccedd000)]
0x00007f85d14e4000 JavaThread "http-bio-8080-exec-712" daemon [_thread_blocked, id=30028, stack(0x00007f85ed6af000,0x00007f85ed7b0000)]
0x00007f85db640800 JavaThread "http-bio-8080-exec-711" daemon [_thread_blocked, id=30027, stack(0x00007f85ed7b0000,0x00007f85ed8b1000)]
0x00007f85dade1000 JavaThread "http-bio-8080-exec-710" daemon [_thread_blocked, id=30026, stack(0x00007f85ed8b1000,0x00007f85ed9b2000)]
0x00007f85d8de8000 JavaThread "http-bio-8080-exec-709" daemon [_thread_blocked, id=30025, stack(0x00007f86000e8000,0x00007f86001e9000)]
0x00007f85d0a29800 JavaThread "http-bio-8080-exec-708" daemon [_thread_blocked, id=30024, stack(0x00007f85c74b9000,0x00007f85c75ba000)]
0x00007f85da827000 JavaThread "http-bio-8080-exec-707" daemon [_thread_blocked, id=30023, stack(0x00007f85c7dc2000,0x00007f85c7ec3000)]
0x00007f85d946b000 JavaThread "http-bio-8080-exec-706" daemon [_thread_blocked, id=30022, stack(0x00007f85c96d1000,0x00007f85c97d2000)]
0x00007f85d17a6800 JavaThread "http-bio-8080-exec-705" daemon [_thread_blocked, id=30021, stack(0x00007f85c5297000,0x00007f85c5398000)]
0x00007f85d9a83000 JavaThread "http-bio-8080-exec-704" daemon [_thread_blocked, id=30020, stack(0x00007f85c6aaf000,0x00007f85c6bb0000)]
0x00007f85d18ff800 JavaThread "http-bio-8080-exec-703" daemon [_thread_blocked, id=30019, stack(0x00007f85c8dc8000,0x00007f85c8ec9000)]
0x00007f85d293a000 JavaThread "http-bio-8080-exec-698" daemon [_thread_blocked, id=30018, stack(0x00007f85ca7e2000,0x00007f85ca8e3000)]
0x00007f85da19b800 JavaThread "http-bio-8080-exec-697" daemon [_thread_blocked, id=30007, stack(0x00007f85c76bb000,0x00007f85c77bc000)]
0x00007f85dbd89000 JavaThread "http-bio-8080-exec-696" daemon [_thread_blocked, id=30005, stack(0x00007f85c70b5000,0x00007f85c71b6000)]
0x00007f85daea1800 JavaThread "http-bio-8080-exec-695" daemon [_thread_blocked, id=30003, stack(0x00007f85c3e83000,0x00007f85c3f84000)]
0x00007f85d07a3000 JavaThread "http-bio-8080-exec-694" daemon [_thread_blocked, id=30002, stack(0x00007f85cb4ed000,0x00007f85cb5ee000)]
0x00007f85dbaa7800 JavaThread "http-bio-8080-exec-693" daemon [_thread_blocked, id=30001, stack(0x00007f85c357a000,0x00007f85c367b000)]
0x00007f85dbe7c000 JavaThread "http-bio-8080-exec-692" daemon [_thread_blocked, id=30000, stack(0x00007f85cd0f3000,0x00007f85cd1f4000)]
0x00007f85dbbd3000 JavaThread "http-bio-8080-exec-691" daemon [_thread_blocked, id=29999, stack(0x00007f85c3b80000,0x00007f85c3c81000)]
0x00007f85d82ab800 JavaThread "http-bio-8080-exec-690" daemon [_thread_blocked, id=29998, stack(0x00007f85c8ec9000,0x00007f85c8fca000)]
0x00007f85d06b3000 JavaThread "http-bio-8080-exec-689" daemon [_thread_blocked, id=29997, stack(0x00007f85c3a7f000,0x00007f85c3b80000)]
0x00007f85db4c9000 JavaThread "http-bio-8080-exec-688" daemon [_thread_blocked, id=29996, stack(0x00007f85c9edb000,0x00007f85c9fdc000)]
0x00007f85db8fd000 JavaThread "http-bio-8080-exec-687" daemon [_thread_blocked, id=29995, stack(0x00007f85c73b8000,0x00007f85c74b9000)]
0x00007f85d86fa800 JavaThread "http-bio-8080-exec-686" daemon [_thread_blocked, id=29994, stack(0x00007f85c3479000,0x00007f85c357a000)]
0x00007f85d071f000 JavaThread "http-bio-8080-exec-685" daemon [_thread_blocked, id=29993, stack(0x00007f85c72b7000,0x00007f85c73b8000)]
0x00007f85d0fb2000 JavaThread "http-bio-8080-exec-684" daemon [_thread_blocked, id=29992, stack(0x00007f85cc6e7000,0x00007f85cc7e8000)]
0x00007f85da5b4800 JavaThread "http-bio-8080-exec-683" daemon [_thread_blocked, id=29991, stack(0x00007f85c6bb0000,0x00007f85c6cb1000)]
0x00007f85da8cb000 JavaThread "http-bio-8080-exec-682" daemon [_thread_blocked, id=29990, stack(0x00007f85c5ca1000,0x00007f85c5da2000)]
0x00007f85d8f6e000 JavaThread "http-bio-8080-exec-677" daemon [_thread_blocked, id=29988, stack(0x00007f85c77bc000,0x00007f85c78bd000)]
0x00007f85d19fe000 JavaThread "http-bio-8080-exec-681" daemon [_thread_blocked, id=29987, stack(0x00007f85c93ce000,0x00007f85c94cf000)]
0x00007f85d0fa5000 JavaThread "http-bio-8080-exec-680" daemon [_thread_blocked, id=29986, stack(0x00007f85c387d000,0x00007f85c397e000)]
0x00007f85d019f000 JavaThread "http-bio-8080-exec-679" daemon [_thread_blocked, id=29985, stack(0x00007f85c3d82000,0x00007f85c3e83000)]
0x00007f85d8b24000 JavaThread "http-bio-8080-exec-678" daemon [_thread_blocked, id=29984, stack(0x00007f85c4a8f000,0x00007f85c4b90000)]
0x00007f85d1e6a800 JavaThread "http-bio-8080-exec-676" daemon [_thread_blocked, id=29983, stack(0x00007f85cbde0000,0x00007f85cbee1000)]
0x00007f85d9f81000 JavaThread "http-bio-8080-exec-675" daemon [_thread_blocked, id=29981, stack(0x00007f85c92cd000,0x00007f85c93ce000)]
0x00007f85daddf800 JavaThread "http-bio-8080-exec-674" daemon [_thread_blocked, id=29978, stack(0x00007f85c69ae000,0x00007f85c6aaf000)]
0x00007f85d9abd000 JavaThread "http-bio-8080-exec-670" daemon [_thread_blocked, id=29977, stack(0x00007f85c599e000,0x00007f85c5a9f000)]
0x00007f85d9a08000 JavaThread "http-bio-8080-exec-671" daemon [_thread_blocked, id=29976, stack(0x00007f85c5a9f000,0x00007f85c5ba0000)]
0x00007f85db357800 JavaThread "http-bio-8080-exec-672" daemon [_thread_blocked, id=29975, stack(0x00007f85c4b90000,0x00007f85c4c91000)]
0x00007f85d911a800 JavaThread "http-bio-8080-exec-673" daemon [_thread_blocked, id=29974, stack(0x00007f85c80c5000,0x00007f85c81c6000)]
0x00007f85d8802000 JavaThread "http-bio-8080-exec-669" daemon [_thread_blocked, id=29973, stack(0x00007f85c5196000,0x00007f85c5297000)]
0x00007f85db4c9800 JavaThread "pool-36-thread-1" [_thread_blocked, id=29961, stack(0x00007f85c61a6000,0x00007f85c62a7000)]
0x00007f85d8226000 JavaThread "http-bio-8080-exec-662" daemon [_thread_blocked, id=29954, stack(0x00007f85c78bd000,0x00007f85c79be000)]
0x00007f85dadcf000 JavaThread "http-bio-8080-exec-663" daemon [_thread_blocked, id=29953, stack(0x00007f85c397e000,0x00007f85c3a7f000)]
0x00007f85d9d7d800 JavaThread "http-bio-8080-exec-664" daemon [_thread_blocked, id=29952, stack(0x00007f85ce6ed000,0x00007f85ce7ee000)]
0x00007f85dbe7a800 JavaThread "http-bio-8080-exec-665" daemon [_thread_blocked, id=29951, stack(0x00007f85c5da2000,0x00007f85c5ea3000)]
0x00007f85d8635000 JavaThread "http-bio-8080-exec-667" daemon [_thread_blocked, id=29949, stack(0x00007f85c3f84000,0x00007f85c4085000)]
0x00007f85daa7b800 JavaThread "http-bio-8080-exec-661" daemon [_thread_blocked, id=29948, stack(0x00007f85c90cb000,0x00007f85c91cc000)]
0x00007f85daa44800 JavaThread "http-bio-8080-exec-644" daemon [_thread_blocked, id=29910, stack(0x00007f85cbfe2000,0x00007f85cc0e3000)]
0x00007f85db40c800 JavaThread "http-bio-8080-exec-649" daemon [_thread_blocked, id=29895, stack(0x00007f85c367b000,0x00007f85c377c000)]
0x00007f85db4d4800 JavaThread "http-bio-8080-exec-648" daemon [_thread_blocked, id=29894, stack(0x00007f85cd6f7000,0x00007f85cd7f8000)]
0x00007f85d04f4800 JavaThread "http-bio-8080-exec-646" daemon [_thread_blocked, id=29893, stack(0x00007f85cc1e4000,0x00007f85cc2e5000)]
0x00007f85d90cb000 JavaThread "http-bio-8080-exec-642" daemon [_thread_blocked, id=29890, stack(0x00007f85cb3ec000,0x00007f85cb4ed000)]
0x00007f85db40b000 JavaThread "http-bio-8080-exec-637" daemon [_thread_blocked, id=29885, stack(0x00007f85c71b6000,0x00007f85c72b7000)]
7f861a0e3000-7f861a0e4000 r--s 00001000 ca:01 310384983 /usr/local/tomcat/lib/tomcat-api.jar
7f861a0e4000-7f861a0e6000 r--s 00001000 ca:01 327158654 /usr/local/tomcat/temp/newrelic-bootstrap5567724053540453581.jar
7f861a0e6000-7f861a0ea000 r--s 000b0000 ca:01 210043060 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/jsse.jar
7f861a0ea000-7f861a1bb000 rw-p 00000000 00:00 0
7f861a1bb000-7f861a1c3000 rw-s 00000000 ca:01 402669995 /tmp/hsperfdata_tomcat/16209
7f861a1c3000-7f861a1c6000 ---p 00000000 00:00 0
7f861a1c6000-7f861a2c9000 rw-p 00000000 00:00 0 [stack:16210]
7f861a2c9000-7f861a2ca000 r--s 00000000 ca:01 327158655 /usr/local/tomcat/temp/instrumentation4280289022478804408.jar
7f861a2ca000-7f861a2cb000 r--s 00003000 ca:01 310384972 /usr/local/tomcat/lib/annotations-api.jar
7f861a2cb000-7f861a2cd000 r--s 00008000 ca:01 293631274 /usr/local/tomcat/bin/tomcat-juli.jar
7f861a2cd000-7f861a2ce000 r--s 00005000 ca:01 293631268 /usr/local/tomcat/bin/commons-daemon.jar
7f861a2ce000-7f861a2cf000 r--s 00006000 ca:01 293631264 /usr/local/tomcat/bin/bootstrap.jar
7f861a2cf000-7f861a2d0000 rw-p 00000000 00:00 0
7f861a2d0000-7f861a2d1000 ---p 00000000 00:00 0
7f861a2d1000-7f861a2d2000 rw-p 00000000 00:00 0
7f861a2d2000-7f861a2d3000 r--p 00021000 ca:01 1303 /usr/lib64/ld-2.17.so
7f861a2d3000-7f861a2d4000 rw-p 00022000 ca:01 1303 /usr/lib64/ld-2.17.so
7f861a2d4000-7f861a2d5000 rw-p 00000000 00:00 0
7ffe366cd000-7ffe366ee000 rw-p 00000000 00:00 0 [stack]
7ffe3670c000-7ffe3670e000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
VM Arguments:
jvm_args: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx1024m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -DOVERRIDE_COMPILED_CACHE=true -DclientUrl=https://katerra-dev.halosys.com/client-api -Dcom.halomem.redis=false -DDB_SERVER=localhost -DDB_PORT=5432 -DDB_NAME=halomem -DDB_USER=halosys -DDB_PASSWORD=halosys -DDB_MAX_POOL=30 -DCONFIG_FILE=classpath:/hm.properties -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/dump/ -javaagent:/usr/local/tomcat/newrelic/newrelic.jar -Djava.endorsed.dirs=/usr/local/tomcat/endorsed -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp
java_command: org.apache.catalina.startup.Bootstrap start
java_class_path (initial): /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar:/usr/local/tomcat/newrelic/newrelic.jar
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0xa09060], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xa09060], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x867a40], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x867c50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: [libjvm.so+0x867c50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x867c50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
--------------- S Y S T E M ---------------
OS:CentOS Linux release 7.1.1503 (Core)
uname:Linux 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 4096, NOFILE 26384, AS infinity
load average:0.40 0.10 0.07
/proc/meminfo:
MemTotal: 1883272 kB
MemFree: 81416 kB
MemAvailable: 68368 kB
Buffers: 0 kB
Cached: 249212 kB
SwapCached: 0 kB
Active: 1592104 kB
Inactive: 101724 kB
Active(anon): 1559520 kB
Inactive(anon): 68704 kB
Active(file): 32584 kB
Inactive(file): 33020 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 108 kB
Writeback: 0 kB
AnonPages: 1444672 kB
Mapped: 130564 kB
Shmem: 183536 kB
Slab: 51212 kB
SReclaimable: 20916 kB
SUnreclaim: 30296 kB
KernelStack: 7328 kB
PageTables: 16912 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 941636 kB
Committed_AS: 3306632 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 5220 kB
VmallocChunk: 34359732223 kB
HardwareCorrupted: 0 kB
AnonHugePages: 628736 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 40960 kB
DirectMap2M: 2056192 kB
CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping : 2
microcode : 0x36
cpu MHz : 2400.042
cache size : 30720 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm xsaveopt fsgsbase bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4800.08
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 1883272k(81416k free), swap 0k(0k free)
vm_info: OpenJDK 64-Bit Server VM (25.71-b15) for linux-amd64 JRE (1.8.0_71-b15), built on Jan 21 2016 16:00:19 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-4)
time: Tue Apr 11 10:44:59 2017
elapsed time: 101403 seconds (1d 4h 10m 3s)
答案 0 :(得分：0)
不会因内存不足相关的错误而生成堆转储。您需要使用-XX：+ HeapDumpOnOutOfMemoryError来获取heapdump，前提是您已通过＆＃34; ulimit -c unlimited＆＃34;选项
将限制设置为无限制