Java性能问题 - Tomcat WebappClassLoader被锁定

时间:2012-10-04 13:54:33

标签: java json performance struts tomcat7

在我做产品性能测试的过程中,我一直面临这个奇怪的问题,我使用的是java 6,struts 3框架和tomcat 7服务器

在性能测试期间,我们开始加载数千个UI请求,最初命中服务器它运行良好但几个小时后请求开始被阻止,然后CPU使用率达到100%甚至UI变得无法访问。我采取了线程转储来分析下面的问题是我一直得到的。

""http-nio-8443"-exec-1305" daemon prio=10 tid=0x00007f0458538000 nid=0x56dd waiting for monitor entry [0x00007f04345c9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1663)
    - locked <0x00000000a162f0c0> (a org.apache.catalina.loader.WebappClassLoader)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1521)
    at java.beans.Introspector.instantiate(Introspector.java:1448)
    at java.beans.Introspector.findExplicitBeanInfo(Introspector.java:431)
    at java.beans.Introspector. (Introspector.java:380)
    at java.beans.Introspector.getBeanInfo(Introspector.java:232)
    at java.beans.Introspector.getBeanInfo(Introspector.java:218)
    at com.googlecode.jsonplugin.JSONWriter.bean(JSONWriter.java:169)
    at com.googlecode.jsonplugin.JSONWriter.process(JSONWriter.java:152)
    at com.googlecode.jsonplugin.JSONWriter.value(JSONWriter.java:120)
    at com.googlecode.jsonplugin.JSONWriter.write(JSONWriter.java:88)
    at com.googlecode.jsonplugin.JSONUtil.serialize(JSONUtil.java:90)
    at com.googlecode.jsonplugin.JSONResult.execute(JSONResult.java:119)
    at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:348)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
    at com.facetime.imauditor.sreach.action.CustomRequestInterceptorJson.intercept(CustomRequestInterceptorJson.java:69)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150)
    at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:123)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:167)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:105)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:83)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:207)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:74)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:127)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:107)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:206)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:115)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:143)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:121)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:170)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:123)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
    at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
    at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
    at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:50)
    at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:504)
    at org.apache.struts2.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:419)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at com.facetime.imcoreserver.registration.AbsoluteSendRedirectFilter.doFilter(AbsoluteSendRedirectFilter.java:48)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403)
    at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:369)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:317)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1532)
    - locked <0x00000000abb53778> (a org.apache.tomcat.util.net.SecureNioChannel)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

以下是线程的统计信息。

Threads locking monitor=1
Threads sleeping on monitor=0
Threads waiting to lock monitor=1097

提前感谢,这方面的任何帮助都会有所帮助。

4 个答案:

答案 0 :(得分:3)

这是struts2 json插件中的一个错误:

https://issues.apache.org/jira/browse/WW-3902

答案 1 :(得分:1)

<强>更新

我最近遇到了类似的问题。有时,当Web应用程序编程不好时,它可能无法正确清理网络(TCP)套接字...例如数据库连接类。导致主机(Tomcat JVM)进程上有太多打开的文件描述符。这可能会导致你面临同样的责任。

因此,在我的情况下,我通过使用ThreadLocal泛型在每个线程的基础上设计数据库连接来解决问题。这是在我的Web-Apps服务托管bean中管理的。:

private static ThreadLocal<YourConnectionClass> connection = new ThreadLocal<YourConnectionClass>();

public static YourConnectionClass getConnection(){
    if (connection.get() == null) connection.set(driver.getConnection());
    return connection.get();
}

这将确保每个tomcat执行程序线程只打开一个TCP套接字。如果这个问题适用于你,你给我提到的那个添加了类似的修复,那么你的问题就会消失。好吧,但它仍然只是一个猜测&#34;。只需将其输入到服务器unix / linux控制台:

,Tou就可以验证这是否适用于您
netstat | grep <YOUR_TCP_PORT>

或使用lsof(列出打开的文件)。因为TCP套接字像打开文件句柄一样处理。

lsof <YOUR_JVM_PID>

如果tomcat启动后打开套接字的数量少或少,而另一方面,当您的责任问题发生时,破坏性很高,那么您可能有一个打开的套接字文件描述符泄漏。

作为第二个猜测,你可能还有一个HashTable内存泄漏,即使在Java中没有正确清理HashTable的Java对象(在web-apps服务器代码中的某个地方)也可能导致内存泄漏。但正如我在上述评论中所读到的那样,可能并非如此。

欢呼声,

<小时/> 我的老答案:

我打赌它的垃圾收集策略。

通过设置正确的JVM OPTS尝试使用另一个:

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:SurvivorRatio=2 -XX:NewRatio=8

检查此链接:

http://www.petefreitag.com/articles/gctuning/

http://www.idevelopment.info/data/Programming/java/miscellaneous_java/Java_Options_For_Garbage_Collection.html

答案 2 :(得分:1)

可以通过升级到Tomcat 8.0.16或更高版本并将Loader类设置为org.apache.catalina.loader.ParallelWebappClassLoader来解决此问题。

在[TOMCAT_HOME] /conf/context.xml中,添加以下行并重新启动

<Loader loaderClass="org.apache.catalina.loader.ParallelWebappClassLoader" />

我们遇到了类似的问题,Hibernate和Spring JDBC会尝试将db结果集行转换为对象。这种锁定行为严重影响了运行大量结果集的线程的tomcats。

我建议用户在Tomcat7上运行Hibernate和Spring JDBC应用程序到Tomcat 8,并启用ParallelWebappClassLoader以获得更好的性能。

答案 3 :(得分:0)

由于在WebappClassLoader上进行了同步,我发现了类似的问题。任何时候都会出现大量的loadClass调用 - 很多第三方库很遗憾。理想的解决方法是不要一直调用loadClass,但这可能不是一个选项。

这源于Tomcat作为bug 48903/44041/48694的一部分引入的更改,以修复一些用户遇到的死锁。

http://svn.apache.org/viewvc?view=revision&revision=927565

在我看来,修复过程对同步过于悲观,希望我能接受一个允许在进入监视器之前检查本地缓存的修复;应该让它消失。