Quartz Scheduler worker threads not running jobs even with huge number of jobs in queue

Question

We have 30 worker threads in quartz scheduler, and there are 104k jobs in the queue (they were piled up since quartz was not processing fast enough and the lots of bulk jobs were submitted). As you can see in the thread dump, most of these threads are just waiting without working. Why are these worker threads not running all those jobs which are in the queue missing the fire time?

Could there be any bottlenecks before the jobs are distributed to the workers? We see only a few jobs running at any time instead of upto 30 jobs at once.

I should note that this happens irrespective of only one scheduler process or many.

Any pointers to find why many (at most 30) jobs are not getting picked up and executed at once would be really appreciated.

Quartz version: 2.1.3 (with Oracle JDBC jobstore)

Config File

#============================================================================
# Configure Main Scheduler Properties
#============================================================================

org.quartz.scheduler.instanceName = ****_SCHEDULER
org.quartz.scheduler.instanceId = AUTO

#============================================================================
# Configure ThreadPool
#============================================================================

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 30
org.quartz.threadPool.threadPriority = 5


#============================================================================
# Configure JobStore
#============================================================================

org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.dataSource = myDS
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval = 20000
org.quartz.jobStore.tablePrefix = QRTZ_


#============================================================================
# Configure Datasources
#============================================================================

org.quartz.dataSource.myDS.driver = oracle.jdbc.driver.OracleDriver
org.quartz.dataSource.myDS.URL = jdbc:oracle:thin:@*****
org.quartz.dataSource.myDS.user = *****
#org.quartz.dataSource.myDS.password = *****
org.quartz.dataSource.myDS.maxConnections = 10
org.quartz.dataSource.myDS.validationQuery=select 0 from dual

#============================================================================
# Configured to batch trigger acquisition
#============================================================================
org.quartz.scheduler.skipUpdateCheck = false
org.quartz.scheduler.batchTriggerAcquisitionMaxCount = 30
org.quartz.jobStore.acquireTriggersWithinLock = true
~

Thread Dump

I see the QuartzSchedulerThread (i assume is like manager thread?) and MisfireHandler threads are always running with same stacktrace (except for locked monitor id).

Stacktrace for QuartzSchedulerThread

at java.net.SocketInputStream.socketRead0(Native Method) 
at java.net.SocketInputStream.read(SocketInputStream.java:129) 
at oracle.net.ns.Packet.receive(Packet.java:300) 
at oracle.net.ns.DataPacket.receive(DataPacket.java:106) 
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) 
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) 
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) 
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) 
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) 
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) 
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) 
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) 
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) 
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) 
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) 
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044) 
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1199) 
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) 
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) 
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628) 
- locked [0x00000007f5d0fd30] (a oracle.jdbc.driver.T4CConnection) 
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) 
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) 
at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:91) 
at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:115) 
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3803) 
at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggersFired(JobStoreSupport.java:2893) 
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:326)

Stacktrace for MisfireHandler thread:

at java.net.SocketInputStream.socketRead0(Native Method) 
at java.net.SocketInputStream.read(SocketInputStream.java:129) 
at oracle.net.ns.Packet.receive(Packet.java:300) 
at oracle.net.ns.DataPacket.receive(DataPacket.java:106) 
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) 
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260) 
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185) 
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102) 
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) 
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) 
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) 
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) 
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) 
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) 
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) 
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044) 
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1199) 
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) 
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) 
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628) 
- locked [0x00000007f58c8010] (a oracle.jdbc.driver.T4CConnection) 
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493) 
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) 
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.hasMisfiredTriggersInState(StdJDBCDelegate.java:379) 
at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:946) 
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3187) 
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3946) 
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3967)