问题是:在负载峰值时CPU卡在40%上,请求响应减慢
我有专用的后端API服务器,实际负载约为7,332次/分钟
数据库在专用服务器上,加载正常
这台机器几乎没有IO操作。
12芯x 2 CPU = 24芯
操作系统:OS Linux,4.4.0-98-generic,amd64 / 64(24核)
java版“1.7.0_151”
OpenJDK运行时环境(IcedTea 2.6.11)(7u151-2.6.11-0ubuntu1.14.04.1)
OpenJDK 64位服务器VM(内置24.151-b01,混合模式)
Tomcat 7.0.82
-Xms50g
-Xmx50g
-XX:PermSize=512m
-XX:MaxPermSize=512m
-XX:MaxJavaStackTraceDepth=-1
-Djava.net.preferIPv4Stack=true
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
-XX:+ScavengeBeforeFullGC
-XX:+CMSScavengeBeforeRemark
二手BIO(同样的东西),现在使用NIO连接器。
<Connector port="8080" redirectPort="8443"
protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="2000"
minSpareThreads="20"
acceptCount="200"
acceptorThreadCount="2"
connectionTimeout="20000"
processorCache="-1"
URIEncoding="UTF-8" />
来自JavaMelody的统计数据
Busy threads = 121 / 2,000
Bytes received = 8,679,400,308
Bytes sent = 83,345,586,407
Request count = 6,169,418
Error count = 961
Sum of processing times (ms) = 2,396,325,165
Max processing time (ms) = 4,168,515
Memory: Non heap memory = 275 Mb (Perm Gen, Code Cache),
Buffered memory = 5 Mb,
Loaded classes = 48,952,
Garbage collection time = 1,238,271 ms,
Process cpu time = 197,922,070 ms,
Committed virtual memory = 66,260 Mb,
Free physical memory = 267 Mb,
Total physical memory = 64,395 Mb,
Free swap space = 0 Mb,
Total swap space = 0 Mb
Perm Gen memory: 247 Mb / 512 Mb
Free disk space: 190,719 Mb
无法在测试服务器上重现此信息 我的瓶颈在哪里?
CPU usage chart from JMX
htop stats
更新:
Profiler screenshot TaskQueue.poll() hangs
Profiler screenshot ordered by self time CPU