Spring + Hibernate在数小时后随机冻结

时间：2016-09-20 01:53:46

标签： java spring hibernate spring-mvc tomcat

我正在使用springMVC + spring + hibernate处理一个Web应用程序，并使用Tomcat 8.0.36进行部署，并将代理传递给 nginx 到端口80.它在开始时正常运行，但之后几个小时（通常是10到20）它冻结，无法处理任何请求。

为了说明我检查了几个日志和停机时间的原因。

  • 在 ngnix 的访问日志中，记录传入的请求，http状态为504或499.

  • 在 tomcat 的访问日志中，在停机期间没有记录任何请求，并且该时间周围的请求都已成功处理（状态为200）

  • 在 catalina.out （log4j设置为调试级别）中，没有记录错误。在那里我可以看到应用程序本身在停机期间似乎仍然存在，因为有IdleConnectionHandler保持记录的事情：

  

09：41：33,246 DEBUG IdleConnectionHandler：108 - 检查连接，idleTimeout：1474076493216

我发现问题可能在于hibernate获取JDBC连接并在进一步实验和日志记录后开始事务。

服务器冻结后，不使用数据库的请求将在 catalina.out 中成功完成记录，但不会记录在tomcats的访问日志中。在ngnix的日志中，它失败了，状态为504。（ UPDATE ：从日志中我发现这些请求也在successfully completed DispatcherServlet记录preparing JDBC connection之后开始尝试。我不知道为什么。也许与Spring事件监听器有关。）

另一方面，对于将通过hibernate访问数据库的请求，他们的线程将以23:38:17,211 DEBUG [THREAD ID=http-nio-8080-exec-156] DefaultListableBeanFactory:251 - Returning cached instance of singleton bean 'queryServiceImpl' 23:38:17,211 DEBUG [THREAD ID=http-nio-8080-exec-156] HibernateTransactionManager:380 - Found thread-bound Session [SessionImpl(PersistenceContext[entityKeys=[],collectionKeys=[]];ActionQueue[insertions=org.hibernate.engine.spi.ExecutableList@6059c46 updates=org.hibernate.engine.spi.ExecutableList@4671f5ca deletions=org.hibernate.engine.spi.ExecutableList@2d02684b orphanRemovals=org.hibernate.engine.spi.ExecutableList@3815fcf9 collectionCreations=org.hibernate.engine.spi.ExecutableList@12d8a011 collectionRemovals=org.hibernate.engine.spi.ExecutableList@5b4ec825 collectionUpdates=org.hibernate.engine.spi.ExecutableList@6530a337 collectionQueuedOps=org.hibernate.engine.spi.ExecutableList@72d5795c unresolvedInsertDependencies=UnresolvedEntityInsertActions[]])] for Hibernate transaction 23:38:17,211 DEBUG [THREAD ID=http-nio-8080-exec-156] HibernateTransactionManager:367 - Creating new transaction with name [com.pia.mainWeb.service.impl.QueryServiceImpl.get]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,readOnly; '' 23:38:17,212 DEBUG [THREAD ID=http-nio-8080-exec-156] HibernateTransactionManager:445 - Preparing JDBC Connection of Hibernate Session [SessionImpl(PersistenceContext[entityKeys=[],collectionKeys=[]];ActionQueue[insertions=org.hibernate.engine.spi.ExecutableList@6059c46 updates=org.hibernate.engine.spi.ExecutableList@4671f5ca deletions=org.hibernate.engine.spi.ExecutableList@2d02684b orphanRemovals=org.hibernate.engine.spi.ExecutableList@3815fcf9 collectionCreations=org.hibernate.engine.spi.ExecutableList@12d8a011 collectionRemovals=org.hibernate.engine.spi.ExecutableList@5b4ec825 collectionUpdates=org.hibernate.engine.spi.ExecutableList@6530a337 collectionQueuedOps=org.hibernate.engine.spi.ExecutableList@72d5795c unresolvedInsertDependencies=UnresolvedEntityInsertActions[]])] 结束，如下所示：

transaction begin

如果我将它与服务器正常运行时应该打印的内容进行比较， 下一行应为23:37:52,966 DEBUG [THREAD ID=http-nio-8080-exec-52] TransactionImpl:51 - begin ：

org.springframework.orm.hibernate5.HibernateTransactionManager

然后我提到了if (this.prepareConnection && isSameConnectionForEntireSession(session)) { // We're allowed to change the transaction settings of the JDBC Connection. if (logger.isDebugEnabled()) { logger.debug("Preparing JDBC Connection of Hibernate Session [" + session + "]"); } Connection con = ((SessionImplementor) session).connection(); Integer previousIsolationLevel = DataSourceUtils.prepareConnectionForTransaction(con, definition); txObject.setPreviousIsolationLevel(previousIsolationLevel); if (this.allowResultAccessAfterCompletion && !txObject.isNewSession()) { int currentHoldability = con.getHoldability(); if (currentHoldability != ResultSet.HOLD_CURSORS_OVER_COMMIT) { txObject.setPreviousHoldability(currentHoldability); con.setHoldability(ResultSet.HOLD_CURSORS_OVER_COMMIT); } } } else { // Not allowed to change the transaction settings of the JDBC Connection. if (definition.getIsolationLevel() != TransactionDefinition.ISOLATION_DEFAULT) { // We should set a specific isolation level but are not allowed to... throw new InvalidIsolationLevelException( "HibernateTransactionManager is not allowed to support custom isolation levels: " + "make sure that its 'prepareConnection' flag is on (the default) and that the " + "Hibernate connection release mode is set to 'on_close' (the default for JDBC)."); } if (logger.isDebugEnabled()) { logger.debug("Not preparing JDBC Connection of Hibernate Session [" + session + "]"); } } if (definition.isReadOnly() && txObject.isNewSession()) { // Just set to MANUAL in case of a new Session for this transaction. session.setFlushMode(FlushMode.MANUAL); } if (!definition.isReadOnly() && !txObject.isNewSession()) { // We need AUTO or COMMIT for a non-read-only transaction. FlushMode flushMode = session.getFlushMode(); if (session.getFlushMode().equals(FlushMode.MANUAL)) { session.setFlushMode(FlushMode.AUTO); txObject.getSessionHolder().setPreviousFlushMode(flushMode); } } Transaction hibTx; // Register transaction timeout. int timeout = determineTimeout(definition); if (timeout != TransactionDefinition.TIMEOUT_DEFAULT) { // Use Hibernate's own transaction timeout mechanism on Hibernate 3.1+ // Applies to all statements, also to inserts, updates and deletes! hibTx = session.getTransaction(); hibTx.setTimeout(timeout); hibTx.begin(); // <---- Will print "begin" here } else { // Open a plain Hibernate transaction without specified timeout. hibTx = session.beginTransaction(); // <---- Will print "begin" here } 类并找到了可能出错的地方（第442行到第500行）：

org.apache.commons.dbcp.PoolingDataSource

然而，我仍然无法弄清楚出了什么问题。崩溃似乎是随机的，我找不到在测试环境中重现它的方法。我的服务器依赖项的版本是：

  • Spring and Spring-mvc 4.2.3.RELEASE
  • Hibernate 5.0.2.Final

在我看来，这些线程被某种东西阻挡了。这可能是交易中某个地方的僵局吗？我现在没有设置事务超时（默认为-1），我不确定这是否会导致线程之间的死锁。

任何人都可以帮助我。任何想法或提示将不胜感激。提前谢谢。

更新：

我试图在服务器挂起时转储线程。问题可能就像@ ike3所描述的那样。所有线程都在等待从"http-nio-8080-exec-132" #6824 daemon prio=5 os_prio=0 tid=0x00007f56c4067000 nid=0x1f40 in Object.wait() [0x00007f56a6981000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1126) - locked <0x000000069b573e18> (a org.apache.commons.pool.impl.GenericObjectPool$Latch) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122) at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:382) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:87) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:112) at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:488) at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:447) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) at com.sun.proxy.$Proxy184.onApplicationEvent(Unknown Source) at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy115.onApplicationEvent(Unknown Source) at com.pia.mainWeb.controller.VetController.onApplicationEvent(VetController.java:1327) at com.pia.mainWeb.controller.VetController$$FastClassBySpringCGLIB$$bb2aa9d5.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:718) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:654) at com.pia.mainWeb.controller.VetController$$EnhancerBySpringCGLIB$$cdc9f985.onApplicationEvent(<generated>) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:163) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:136) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:380) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334) at org.springframework.web.servlet.FrameworkServlet.publishRequestHandledEvent(FrameworkServlet.java:1073) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.pia.mainWeb.filter.UrlSessionFilter.doFilter(UrlSessionFilter.java:80) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.pia.mainWeb.filter.ReferralFilter.doFilter(ReferralFilter.java:89) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:87) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.springframework.orm.hibernate5.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:151) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.pia.mainWeb.filter.DomainFilter.doFilter(DomainFilter.java:42) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1099) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:670) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476) - locked <0x00000007639923d0> (a org.apache.tomcat.util.net.NioChannel) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) 获取连接。 

org.apache.tomcat.jdbc.pool.DataSource

但是我仍然不确定解决方案。现在，我将dataSource Implementation更改为testWhileIdle，SELECT 1设置为true，<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource"> <property name="driverClassName" value="${mvcdb.mysql.driver}"/> <property name="validationQuery" value="select 1"/> <property name="validationInterval" value="10000"/> <property name="url" value="${mvcdb.mysql.url}"/> <property name="username" value="${mvcdb.mysql.username}"/> <property name="password" value="${mvcdb.mysql.password}"/> <property name="initialSize" value="${mvcdb.mysql.initialSize}"/> <property name="maxActive" value="${mvcdb.mysql.maxActive}"/> <property name="minIdle" value="${mvcdb.mysql.maxActive}"/> <property name="maxIdle" value="${mvcdb.mysql.maxActive}"/> <property name="maxWait" value="${mvcdb.mysql.maxWait}"/> <property name="testOnBorrow" value="true"/> <property name="testWhileIdle" value="true"/> <property name="removeAbandoned" value="true"/> <property name="removeAbandonedTimeout" value="120"/> </bean> 验证查询。效果还需要进一步检查。 

import groovy.json.*
import hudson.model.*
import jenkins.model.Jenkins

  // get current thread / Executor
      def applicationLatestBuild = getLatestBuild('application')

def getLatestBuild( jobName ) {
  def searchUrl = "http://xyz.nbc.com:9090/api/search/artifact?name=${jobName}&repos=libs-snapshot-local"
  def conn = searchUrl.toURL().openConnection()
  conn.setRequestProperty("X-Result-Detail", "info, properties")
  def searchResultTxt = conn.content.text
  //println "Found: ${searchResultTxt}"
  def searchResults = new JsonSlurper().parseText(searchResultTxt)
  def builds = searchResults.results.findAll{it.properties["build.number"] != null}.collect { Integer.parseInt(it.properties["build.number"][0]) }.sort().unique().reverse()
  builds[0]
}

Result is :

Result:   168

0 个答案:

没有答案
相关问题
最新问题