在我的计算服务器上启动单线程应用程序时,我刚遇到(openJDK)JVM的一个非常奇怪的行为:
JVM启动了很多(非常多)线程,大大增加了所有处理器的CPU负载,我没有合适的解释[见下图]。
我还测试了一些其他内容:
java -Xms4096M -Xmx100000M -jar ...
启动应用,而是尝试使用较小的参数,例如java -Xms1024M -Xmx1024M -jar ...
,但没有任何更改。还有其他人已经经历过这样的行为吗?
甚至更好:有人可以向我解释这里发生了什么吗?
修改
我现在已经尝试了一些你建议的东西,并且确实得到了一些有价值且无价值的输出。
jstack PID
不起作用,建议我使用jstack -F PID
,其主要流程的输出如下。
kill -3 PID
不会输出任何内容。
Attaching to process ID 39041, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.40-b25
Deadlock Detection:
No deadlocks found.
Thread 39090: (state = BLOCKED)
Thread 39089: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Interpreted frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)
Thread 39088: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=36, line=157 (Interpreted frame)
Thread 39043: (state = BLOCKED)
- java.lang.AbstractStringBuilder.<init>(int) @bci=6, line=68 (Compiled frame)
- java.lang.StringBuilder.<init>(java.lang.String) @bci=8, line=112 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=526, line=99 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=413, line=86 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=80, line=39 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=136, line=43 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=676, line=112 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=654, line=111 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=654, line=111 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=654, line=111 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=654, line=111 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=654, line=111 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=654, line=111 (Compiled frame)
- de.unisb.react.stud.algorithm.BMCLoopConverter.convert(gov.nasa.ltl.trans.Formula, int, int, int[]) @bci=654, line=111 (Compiled frame)
- de.unisb.react.stud.algorithm.PMC.LoopConvert(gov.nasa.ltl.trans.Formula, int, de.unisb.react.stud.algorithm.Tree) @bci=69, line=274 (Compiled frame)
- de.unisb.react.stud.algorithm.PMC.main(java.lang.String[]) @bci=207, line=97 (Interpreted frame)
我不确定它是属于kill还是jstack,但是我在tmux中发现了这个:
2015-04-06 10:11:49
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
"Service Thread" #23 daemon prio=9 os_prio=0 tid=0x00007f0e54040000 nid=0x98c5 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread17" #22 daemon prio=9 os_prio=0 tid=0x00007f0e54039800 nid=0x98c4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread16" #21 daemon prio=9 os_prio=0 tid=0x00007f0e54037800 nid=0x98c3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread15" #20 daemon prio=9 os_prio=0 tid=0x00007f0e54035000 nid=0x98c2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f0e54033000 nid=0x98c1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f0e54031000 nid=0x98c0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f0e5402f000 nid=0x98bf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f0e5402d000 nid=0x98be waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f0e5402b000 nid=0x98bd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x0000000001bd9000 nid=0x98bc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x0000000001bd7000 nid=0x98bb waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x0000000001bd5000 nid=0x98ba waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x0000000001bd3000 nid=0x98b9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x0000000001bd1000 nid=0x98b8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x0000000001bc6800 nid=0x98b7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x0000000001bc4800 nid=0x98b6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x0000000001bc0000 nid=0x98b5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x0000000001bbe000 nid=0x98b4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x0000000001bbb800 nid=0x98b3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x0000000001bb9000 nid=0x98b2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x0000000001b89000 nid=0x98b1 in Object.wait() [0x00007f0e5b5ba000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f0f73829c78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00007f0f73829c78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x0000000001b87000 nid=0x98b0 in Object.wait() [0x00007f0e5b6bb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f0f73829cc0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00007f0f73829cc0> (a java.lang.ref.Reference$Lock)
"main" #1 prio=5 os_prio=0 tid=0x000000000151d800 nid=0x9883 runnable [0x00007f27cbc8c000]
java.lang.Thread.State: RUNNABLE
at de.unisb.react.stud.algorithm.BMCLoopConverter.convertLoop(BMCLoopConverter.java:156)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:118)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.BMCLoopConverter.convert(BMCLoopConverter.java:111)
at de.unisb.react.stud.algorithm.PMC.LoopConvert(PMC.java:274)
at de.unisb.react.stud.algorithm.PMC.main(PMC.java:97)
"VM Thread" os_prio=0 tid=0x0000000001b82000 nid=0x98af runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000001533000 nid=0x9884 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000001535000 nid=0x9885 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000001536800 nid=0x9886 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000001538800 nid=0x9887 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x000000000153a000 nid=0x9888 runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x000000000153c000 nid=0x9889 runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x000000000153d800 nid=0x988a runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x000000000153f800 nid=0x988b runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x0000000001541000 nid=0x988c runnable
"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x0000000001543000 nid=0x988d runnable
"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x0000000001544800 nid=0x988e runnable
"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x0000000001546800 nid=0x988f runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x0000000001548000 nid=0x9890 runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x0000000001548000 nid=0x9890 runnable [31/1020]
"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x000000000154a000 nid=0x9891 runnable
"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x000000000154b800 nid=0x9892 runnable
"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x000000000154d800 nid=0x9893 runnable
"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x000000000154f000 nid=0x9894 runnable
"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x0000000001551000 nid=0x9895 runnable
"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x0000000001552800 nid=0x9896 runnable
"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x0000000001554800 nid=0x9897 runnable
"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x0000000001556000 nid=0x9898 runnable
"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x0000000001558000 nid=0x9899 runnable
"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x0000000001559800 nid=0x989a runnable
"GC task thread#23 (ParallelGC)" os_prio=0 tid=0x000000000155b800 nid=0x989b runnable
"GC task thread#24 (ParallelGC)" os_prio=0 tid=0x000000000155d000 nid=0x989c runnable
"GC task thread#25 (ParallelGC)" os_prio=0 tid=0x000000000155f000 nid=0x989d runnable
"GC task thread#26 (ParallelGC)" os_prio=0 tid=0x0000000001560800 nid=0x989e runnable
"GC task thread#27 (ParallelGC)" os_prio=0 tid=0x0000000001562800 nid=0x989f runnable
"GC task thread#28 (ParallelGC)" os_prio=0 tid=0x0000000001564000 nid=0x98a0 runnable
"GC task thread#29 (ParallelGC)" os_prio=0 tid=0x0000000001566000 nid=0x98a1 runnable
"GC task thread#30 (ParallelGC)" os_prio=0 tid=0x0000000001567800 nid=0x98a2 runnable
"GC task thread#31 (ParallelGC)" os_prio=0 tid=0x0000000001569800 nid=0x98a3 runnable
"GC task thread#32 (ParallelGC)" os_prio=0 tid=0x000000000156b000 nid=0x98a4 runnable
"GC task thread#33 (ParallelGC)" os_prio=0 tid=0x000000000156d000 nid=0x98a5 runnable
"GC task thread#34 (ParallelGC)" os_prio=0 tid=0x000000000156e800 nid=0x98a6 runnable
"GC task thread#35 (ParallelGC)" os_prio=0 tid=0x0000000001570800 nid=0x98a7 runnable
"GC task thread#36 (ParallelGC)" os_prio=0 tid=0x0000000001572000 nid=0x98a8 runnable
"GC task thread#37 (ParallelGC)" os_prio=0 tid=0x0000000001574000 nid=0x98a9 runnable
"GC task thread#38 (ParallelGC)" os_prio=0 tid=0x0000000001575800 nid=0x98aa runnable
"GC task thread#39 (ParallelGC)" os_prio=0 tid=0x0000000001577800 nid=0x98ab runnable
"GC task thread#40 (ParallelGC)" os_prio=0 tid=0x0000000001579000 nid=0x98ac runnable
"GC task thread#41 (ParallelGC)" os_prio=0 tid=0x000000000157b000 nid=0x98ad runnable
"GC task thread#42 (ParallelGC)" os_prio=0 tid=0x000000000157c800 nid=0x98ae runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f0e54042800 nid=0x98c6 waiting on condition
JNI global references: 13
Heap [163/1020]
PSYoungGen total 22755328K, used 10392421K [0x00007f1f96300000, 0x00007f27b9800000, 0x00007f27b9800000)
eden space 11377664K, 91% used [0x00007f1f96300000,0x00007f22107d97a8,0x00007f224ca00000)
from space 11377664K, 0% used [0x00007f224ca00000,0x00007f224ca00000,0x00007f2503100000)
to space 11377664K, 0% used [0x00007f2503100000,0x00007f2503100000,0x00007f27b9800000)
ParOldGen total 68267008K, used 68266986K [0x00007f0f4f800000, 0x00007f1f96300000, 0x00007f1f96300000)
object space 68267008K, 99% used [0x00007f0f4f800000,0x00007f1f962fa968,0x00007f1f96300000)
Metaspace used 3156K, capacity 4240K, committed 4352K, reserved 8192K
我检查了生成高CPU负载的正在运行的进程的PID - &gt;它们属于GC线程。 我还不确定单个线程程序如何产生如此多的垃圾,以至于它让所有GC线程都忙碌......但至少这是一个起点。 完全解释和调试代码(由于它是一个研究项目,这有点复杂)显然超出了范围。
可能是关于该主题的最后一个问题:是否可以调整JVM产生的GC线程数?
我现在将开始调试垃圾收集,例如使用-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
,如果您知道有用的工具或有任何提示,请告诉我。 ;)
答案 0 :(得分:1)
我能给你的唯一答案是:
尽管其他答案可能意味着什么,但您确实有证据表明出现了问题。在查看操作系统级线程时,Java线程与操作系统线程之间存在一对一的对应关系......至少在Linux上,与当前的JVM一致。
这不是JVM的错。 Java不会自发地创建大量线程;例如你看到~70岁。
(根据Java版本,JVM参数等,您可能会看到十个左右的内部线程。但是大部分时间它们都将处于空闲状态...不会像监控似乎建议的那样一直运行。)
摆弄Java堆和堆栈大小参数不会影响启动的线程数...以一种可能解决问题的方式。
在不同的平台(EC2)上运行不太可能有所作为。
您需要查看应用程序正在执行的操作以及它正在使用的第三方库。除非您提供更多详细信息,否则我们无法帮助您。
但是,获取线程转储(使用jstack
或kill -3 PID
)应提供一些线索,了解这些线程是什么以及它们是如何创建的。
话虽如此,你似乎确实有一个可疑的大堆,并且(可能)有大量核心。显然,吞吐量收集器的默认策略是为每个物理核心创建一个GC线程。但是,很难想象一个用例,单线程应用程序可以创建足够的垃圾来保持大量GC线程长时间忙碌......除非它已经用可填充的对象填充堆。