我运行25000个客户端，每1秒钟就会将日志上传到服务器。服务器在进程中崩溃。从日志文件中，我们发现崩溃的原因是JVM崩溃。错误日志显示：
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f032964f085, pid=2043, tid=0x00007f02955cd700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x5c4085] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x45
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid2043.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
我的JVM参数和系统以及more infos：
VM Arguments:
jvm_args: -Xms256M -Xmx16G -XX:+UseG1GC -Dfile.encoding=UTF8 -Dserver_log_dir=/var/log/kaa -Dserver_log_sufix= -Dserver_home_dir=/usr/lib/kaa-node -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
java_command: org.kaaproject.kaa.server.node.KaaNodeApplication
java_class_path (initial): /usr/lib/kaa-node/conf:/usr/lib/kaa-node/lib/spring-context-4.2.5.RELEASE.jar:/usr/lib/kaa-node/lib/curator-client-2.9.0.jar:/usr/lib/kaa-node/lib/spring-jdbc-4.2.5.RELEASE.jar:/usr/lib/kaa-node/lib/javax.annotation-api-1.2.jar:/usr/lib/kaa-node/lib/log4j-over-slf4j-1.7.7.jar:/usr/lib/kaa-node/lib/jna-4.0.0.jar:/usr/lib/kaa-node/lib/jetty-server-9.2.2.v20140723.jar:/usr/lib/kaa-node/lib/fastutil-6.5.7.jar:/usr/lib/kaa-node/lib/application-action-0.0.64.jar:/usr/lib/kaa-node/lib/commons-collections-3.2.1.jar:/usr/lib/kaa-node/lib/joda-time-2.2.jar:/usr/lib/kaa-node/lib/httpcore-4.3.2.jar:/usr/lib/kaa-node/lib/velocity-1.7.jar:/usr/lib/kaa-node/lib/spring-tx-4.2.5.RELEASE.jar:/usr/lib/kaa-node/lib/hibernate-entitymanager-4.3.11.Final.jar:/usr/lib/kaa-node/lib/jetty-http-9.2.2.v20140723.jar:/usr/lib/kaa-node/lib/commons-cli-1.2.jar:/usr/lib/kaa-node/lib/gwt-client-0.2.1.jar:/usr/lib/kaa-node/lib/swagger-annotations-1.5.9.jar:/usr/lib/kaa-node/lib/jandex-1.1.0.Final.jar:/usr/lib/kaa-node/lib/core-0.10.0.jar:/usr/lib/kaa-node/lib/jetty-security-9.2.2.v20140723.jar:/usr/lib/kaa-node/lib/commons-compress-1.8.jar:/usr/lib/kaa-node/lib/jackson-core-asl-1.9.13.jar:/usr/lib/kaa-node/lib/netty-codec-4.0.34.Final.jar:/usr/lib/kaa-node/lib/dao-0.10.0.jar:/usr/lib/kaa-node/lib/file-appender-0.10.0.jar:/usr/lib/kaa-node/lib/spring-security-web-3.2.9.RELEASE.jar:/usr/lib/kaa-node/lib/gwtquery-1.4.2.jar:/usr/lib/kaa-node/lib/facebook-verifier-0.10.0.jar:/usr/lib/kaa-node/lib/transport-0.10.0-tcp.jar:/usr/lib/kaa-node/lib/spring-data-mongodb-1.9.4.RELEASE.jar:/usr/lib/kaa-node/lib/cassandra-driver-extras-3.0.0.jar:/usr/lib/kaa-node/lib/cassandra-all-3.4.jar:/usr/lib/kaa-node/lib/jcl-over-slf4j-1.7.21.jar:/usr/lib/kaa-node/lib/ant-launcher-1.9.4.jar:/usr/lib/kaa-node/lib/hamcrest-core-1.3.jar:/usr/lib/kaa-node/lib/aspectjrt-1.7.4.jar:/usr/lib/kaa-node/lib/guava-18.0.jar:/usr/lib/kaa-node/lib/spring-beans-4.2.5.RELEASE.jar:/usr/lib/kaa-node/lib/kaa-node-0.10
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL=/bin/bash
SYSTEM:
OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
uname:Linux 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64
libc:glibc 2.19 NPTL 2.19
rlimit: STACK 8192k, CORE 0k, NPROC 32768, NOFILE 65536, AS infinity
load average:23.81 21.57 24.39
/proc/meminfo:
MemTotal: 32629180 kB
MemFree: 11245384 kB
MemAvailable: 16204112 kB
Buffers: 116504 kB
Cached: 5084432 kB
SwapCached: 0 kB
Active: 9205152 kB
Inactive: 3234744 kB
Active(anon): 7260192 kB
Inactive(anon): 1048 kB
Active(file): 1944960 kB
Inactive(file): 3233696 kB
Unevictable: 8523068 kB
Mlocked: 8523068 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 3096 kB
Writeback: 0 kB
AnonPages: 15762160 kB
Mapped: 168560 kB
Shmem: 1384 kB
Slab: 280612 kB
SReclaimable: 181816 kB
SUnreclaim: 98796 kB
KernelStack: 17856 kB
PageTables: 36468 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 16314588 kB
Committed_AS: 18242600 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 15398912 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 70872 kB
DirectMap2M: 2754560 kB
DirectMap1G: 30408704 kB
CPU:total 8 (4 cores per cpu, 2 threads per core) family 6 model 60 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3800.109
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3860.859
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3799.968
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3799.968
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3893.906
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3800.109
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3799.968
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
stepping : 3
microcode : 0x1d
cpu MHz : 3799.968
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs :
bogomips : 7183.28
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 32629180k(11245384k free), swap 0k(0k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.111-b14) for linux-amd64 JRE (1.8.0_111-b14), built on Sep 22 2016 16:14:03 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Fri Nov 25 05:01:22 2016
elapsed time: 21112 seconds (0d 5h 51m 52s)
我对JVM的经验有限。所以我在网上搜索了很长时间，并在Oracle site找到了相关的错误。但我没有找到解决方案。从我的错误日志：
内存：4k页，物理32629180k（11245384k免费），交换0k（0k免费）
显示物理内存占用太多。这可能是由任何破坏堆内存的错误引起的。 GC可能是一个问题，编译器带有错误的本机代码。
答案 0 :(得分：4)
如果您不使用可能已损坏堆的任何本机库，则这是JVM中的错误。您应该检查Oracle是否已经知道该错误，并且（如果没有）提交错误报告。
有问题的框架（G1ParScanThreadState::copy_to_survivor_space）的名称强烈暗示垃圾收集器（GC）崩溃。因此，对于解决方法，直到修复错误，您可以尝试以下任何一种方法：
当您尝试解决某个错误时，它会反复尝试。