我尝试追踪从ColdFusion 9升级到11后间歇性地遇到的一些性能问题.JVM版本从1.6 / 1.7变为1.8(目前是U40)
服务器运行正常一周左右,然后经历性能下降期,直到重新启动。这些时段持续30秒到一分钟,每隔几分钟发生一次。持续时间延长,中间时间随着时间的推移而减少。重新启动后,性能恢复正常
在性能下降时采集的堆栈跟踪表明有很多线程"等待条件"但是我看不到这种情况。通常,我希望看到线程在堆栈跟踪顶部等待的原因。
在这种情况下,我们没有。
查看@ GC日志,这似乎与STW GC集合无关。 (它们大约每30分钟到每小时一次)
堆栈跟踪采取2秒部分显示线程正在移动到代码的不同部分,只是为了击中"等待条件"再次。实际上,相同的请求(线程)可以通过它的可观察生命来移动等待条件。
如果我在服务器上投入大量负载,当它运行良好时,我就不会看到任何"等待条件"在工人线程中。
这是一个示例堆栈跟踪。
" AJP-BIO-8014-EXEC-281" #70085 daemon prio = 5 os_prio = 0 tid = 0x0000000074c73000 nid = 0x21b0等待条件[0x00000000487cb000] java.lang.Thread.State:RUNNABLE 在cfCPStandard2ecfc1872558760 $ funcGETMAINCONTENTHEADER.runFunction(D:\ JO3 \ jobOffice \ sys \ jo \ core \ de \ templates \ cp \ CPStandard.cfc) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2845) 在cfCPStandard2ecfc1872558760 $ funcLOADPAGEHEADER.runFunction(D:\ JO3 \ jobOffice \ sys \ jo \ core \ de \ templates \ cp \ CPStandard.cfc:54) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:643) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:432) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:402) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2483) 在cfDisplayEngine2ecfc1810328208 $ funcPOSTPROCESS.runFunction(D:\ JO3 \ jobOffice \ sys \ jo \ core \ m2plugin \ DisplayEngine.cfc:52) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:643) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:432) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:402) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2483) 在cfPluginManager2ecfc1495337580 $ funcPOSTPROCESS.runFunction(D:\ JO3 \ jobOffice \ MachII \ framework \ PluginManager.cfc:591) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:643) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:432) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:402) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2483) 在cfRequestHandler2ecfc1676430457 $ funcPROCESSEVENTS.runFunction(D:\ JO3 \ jobOffice \ MachII \ framework \ RequestHandler.cfc:308) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2845) 在cfRequestHandler2ecfc1676430457 $ funcHANDLEREQUEST.runFunction(D:\ JO3 \ jobOffice \ MachII \ framework \ RequestHandler.cfc:208) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:643) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:432) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:402) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2483) 在cfmach2dii2ecfc1440816939 $ funcHANDLEREQUEST.runFunction(D:\ JO3 \ jobOffice \ MachII \ mach-ii.cfc:210) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2845) 在cfmach2dii2ecfc1440816939 $ funcONREQUESTSTART.runFunction(D:\ JO3 \ jobOffice \ MachII \ mach-ii.cfc:121) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2845) at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2503) 在cfApplication2ecfc1530162842 $ funcONREQUESTSTART.runFunction(D:\ JO3 \ wwwroot \ Application.cfc:233) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487) at coldfusion.runtime.UDFMethod $ ReturnTypeFilter.invoke(UDFMethod.java:420) at coldfusion.runtime.UDFMethod $ ArgumentCollectionFilter.invoke(UDFMethod.java:383) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:643) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:432) at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:402) at coldfusion.runtime.AppEventInvoker.invoke(AppEventInvoker.java:108) at coldfusion.runtime.AppEventInvoker.onRequestStart(AppEventInvoker.java:279) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:455) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:142) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) 在coldfusion.CfmServlet.service(CfmServlet.java:219) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) 在org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) 在org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) 在org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) 在org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在java.lang.reflect.Method.invoke(Method.java:497) at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:97) at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:472) at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:312) at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:192) at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:507) at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36) at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在java.lang.reflect.Method.invoke(Method.java:497) 在com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:79) at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在java.lang.reflect.Method.invoke(Method.java:497) at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:53) at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut $ 1.invoke(NewFilterChainPointCut.java:41) 在org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java) 在org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) 在org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) 在org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 在org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) 在org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:422) 在org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:199) at org.apache.coyote.AbstractProtocol $ AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint $ SocketProcessor.run(JIoEndpoint.java:314) - 已锁定< 0x000000051d486850> (a org.apache.tomcat.util.net.SocketWrapper) 在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread $ WrappingRunnable.run(TaskThread.java:61) 在java.lang.Thread.run(Thread.java:745)
我的问题是:如何追踪[0x00000000487cb000]是什么?它似乎来自于JVM本身。 (即,不是用户代码生成的等待)。