long running @Scheduled Job lead to DB connection leakage

时间:2018-08-22 14:00:58

标签: spring-boot connection-pooling hikaricp

We have a Springboot 1.5 Application. There are multiple @Scheduled jobs inside @Transactional Services, which may run for multiple minutes. In these jobs, JPA is used to read/write entities -> DB-Connections are used. The

Since we have a lot of exceptions like below

2018-08-22T02:00:36.651 ERROR [ThreadPoolTaskScheduler-8] TaskUtils$LoggingErrorHandler: Unexpected error occurred in scheduled task.
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:377)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:461)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
    at ch.puzzle.b4u.price.control.UpdatePriceService$$EnhancerBySpringCGLIB$$c1835777.updateDynamicPrices(<generated>)
    at sun.reflect.GeneratedMethodAccessor422.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
    at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)

I enabled leakDetection.

The whole datasource config:

datasource:
    type: com.zaxxer.hikari.HikariDataSource
    hikari:
        minimumIdle: 10
        maximumPoolSize: 30
        idleTimeout: 10000
        poolName: SpringBootJPAHikariCP
        maxLifetime: 2000000
        connectionTimeout: 10000
        leakDetectionThreshold: 60000

So after one minute (yes, Threshold is 60000) in a @Scheduled Job, I get now

2018-08-22T07:16:10.007 WARN  [SpringBootJPAHikariCP housekeeper] ProxyLeakTask: Connection leak detection triggered for org.postgresql.jdbc.PgConnection@3b0ca5e1 on thread ThreadPoolTaskScheduler-2, stack trace follows
java.lang.Exception: Apparent connection leak detected
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
    at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
    at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:106)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:136)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:254)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:262)

This connection leakage leads to big problems: As soon as the connections from the pool all are leaked, every further function, which uses the DB, is blocked for a given time (I do not clearly understand why and after which time the connections are working again).

So what should be done? Is it essential, whether we annotate the @Scheduled also @Async? Do we need to set a longer @Transaction timeout than default (I do not know what the default is - and I do not see errors showing that the java transaction is aborted). Any help?

0 个答案:

没有答案