WildFly - HornetQ连接超时和性能问题

时间：2017-05-16 11:48:33

标签： java-ee transactions wildfly hornetq

在WildFly 8.2.1-Final高容量安装中，我们在大约一周内遇到性能下降，直到需要重新启动以及由于状态ABORT_ONLY中的事务而发送JMS消息的次数越来越频繁：

Caused by: javax.ejb.EJBException: javax.jms.JMSException: Could not create a session: IJ000457: Unchecked throwable in managedConnectionReconnected()
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3915e409[state=NORMAL managed connection=org.hornetq.ra.HornetQRAManagedConnection@7d6bcd06 connection handles=0 lastUse=1494579251303 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri@691085ec mcp=SemaphoreArrayListManagedConnectionPool@66a127f5[pool=HornetQConnectionDefinition] xaResource=XAResourceWrapperImpl@56a9bb4[xaResource=org.hornetq.ra.HornetQRAXAResource@4ca920eb pad=false overrideRmValue=null productName=HornetQ productVersion=2.0 jndiName=java:/JmsXA] txSync=null]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:780)
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:518)
        at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:948)
        ... 261 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:551)
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:775)
        ... 263 more
Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff0a0001c3:682851c6:5900737b:dd09939 status: ActionStatus.ABORT_ONLY >
        at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:848)
        at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:383)
        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:544)
        ... 264 more

事务处于ABORT_ONLY状态的原因是之前记录的警告：

2017-05-30 11:33:15,977 WARN  [com.arjuna.ats.jta] (Thread-123) ARJUNA016086: TransactionImple.enlistResource setTransactionTimeout on XAResource < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a0001c3:-7a698606:591fe1ff:82cea51, node_name=1, branch_uid=0:ffff0a0001c3:-7a698606:591fe1ff:830e516, subordinatenodename=null, eis_name=java:/JmsXA > threw: XAException.XAER_RMERR: javax.transaction.xa.XAException
Caused by: HornetQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 51]
       at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:390)
       at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1901)

一些细节：

  • 获得超时的两种数据包类型是51（SESS_XA_START）和63（SESS_XA_SET_TIMEOUT）
  • 该事务中唯一的其他参与者是Oracle XA连接
  • 有消息正在消耗的0个队列和1个（持久的）主题（消息计数在0或0以上）

消息是从单例EJB发送的，而相关的业务方法是用@Lock(LockType.READ)注释的，因此可以同时调用它。 EJB注入java:/JmsXA TopicConnectionFactory和Topic，并在每次调用方法时创建一个新的TopicConnection，Session和MessageProducer（每个消息被发送）并注意在finally {}块中关闭所有消息。

由于超时发送数据包51（SESS_XA_START）和63（SESS_XA_SET_TIMEOUT），似乎开始会话很慢或挂起。

知道原因是什么吗？

2 个答案:

答案 0 :(得分：1)

由于我们可以看到在成千上万的这些错误中只涉及几个不同的连接，我们怀疑连接以某种方式破坏并留在池中，因此这些断开的连接被重用，导致连续错误。

此similar issue中的第5个帖子说：

  

JmsXA连接工厂的连接确实来自   一个游泳池。但是，每个连接都附加了一个侦听器   因此，当发生连接级错误时，应该踢它   JCA实现自动退出池。该   关于你的问题最奇怪的是，一切都是本地的，所以那里   不应该是任何断开的连接或任何连接问题   任何形式。

为了确保从池中删除这些断开的连接，我们在{jagamar-core-impl 1.1.9的cl.getManagedConnection().destroy()中的第799行插入了对AbstractConnectionManager#reconnectManagedConnection(ConnectionListener)的调用，替换了JAR并让系统现在跑了几个星期。

到目前为止它运行稳定，不会减速甚至挂起。

添加到同一方法的某些日志记录会记录池统计信息，我们可以看到在此期间100个连接被破坏但更重要的是，在每个初始错误中都涉及单个连接，因此这些断开的连接不会被重用更多。

导致连接断开的初始错误仍然不清楚。从日志记录看，两个线程同时使用相同的JMS连接，弄乱了导致超时的通信，最终使连接处于断开状态。

另一个猜测是，这是Ironjacamar 1.1.9中的一个问题，很久以前就已修复，并且在WildFly的后续版本中不会出现。

答案 1 :(得分：1)

在jboss上使用Ironjacamar 1.0.9进行了同样的问题7.1.1使用JCA PooledConnectionFactory在Mysql和JMS资源上进行XA事务的最终问题（并以同样的方式修补它，见下文）

以下是错误流程：

  • 以Mysql超时开始（JTA协调器超时设置为100秒）

    • ARJUNA012095：操作中止id 0：ffff0af02d68：-63fd0719：5ab027d7：5afb5在其中有多个线程处于活动状态时被调用。
    • TransactionReaper ::检查成功标记为TX 0：ffff0af02d68：-63fd0719：5ab027d7：5afb5仅作为回滚
    • JDBCExceptionReporter - 超出锁定等待超时;尝试重新启动交易

  • 然后出现了第一个XA错误（在另一个主题上）

    • XA操作失败无法启动，会话已在事务中执行XidImpl xxx

  • 最后，每次在不同的线程上反复出现同样的错误，越来越频繁地引用相同的交易ID

    • XA操作失败无法启动，会话已在事务中执行XidImpl xxx

最后，100％的JMS后期操作在同一错误上失败，直到我们重新启动服务器或连接池被刷新并从scatch重新填充

在pach代码下方（根据TorstenRömer回答）

protected void reconnectManagedConnection(ConnectionListener cl) throws ResourceException
   {
      try
      {
         managedConnectionReconnected(cl);
      }
      catch (Throwable t)
      {
         disconnectManagedConnection(cl);
         // begin patch
         try
         {
            log.warn("JCA patch : forcing destroy on disconnected managed connection ");
            cl.getManagedConnection().destroy();
         }
         catch (ResourceException e)
         {
            log.warn("JCA patch : ResourceException thrown while forcing connection destroy : " + e.getMessage());
         }
         // end patch
         throw new ResourceException(bundle.uncheckedThrowableInManagedConnectionReconnected(cl), t);
      }
   }

在具有错误流的原始日志（截断）下面

2018-03-20 05:08:04,408 [] [Transaction Reaper] WARN com.arjuna.ats.arjuna - ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 in state  RUN
2018-03-20 05:08:04,411 [] [Transaction Reaper Worker 0] WARN com.arjuna.ats.arjuna - ARJUNA012095: Abort of action id 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 invoked while multiple threads active within it.
2018-03-20 05:08:04,411 [] [Transaction Reaper Worker 0] WARN com.arjuna.ats.arjuna - ARJUNA012108: CheckedAction::check - atomic action 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 aborting with 1 threads active!
2018-03-20 05:08:04,911 [] [Transaction Reaper] WARN com.arjuna.ats.arjuna - ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 in state  CANCEL
2018-03-20 05:08:05,413 [] [Transaction Reaper] WARN com.arjuna.ats.arjuna - ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 in state  CANCEL_INTERRUPTED
2018-03-20 05:08:05,414 [] [Transaction Reaper] WARN com.arjuna.ats.arjuna - ARJUNA012120: TransactionReaper::check worker Thread[Transaction Reaper Worker 0,5,main] not responding to interrupt when cancelling TX 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 -- worker marked as zombie and TX scheduled for mark-as-rollback
2018-03-20 05:08:05,414 [] [Transaction Reaper] WARN com.arjuna.ats.arjuna - ARJUNA012110: TransactionReaper::check successfuly marked TX 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 as rollback only

[...]

2018-03-20 05:08:26,882 [req.remoteHost=10.240.45.100, tenant=graxxx] [ajp--0_0_0_0_0_0_0_0-8009-23] ERROR o.h.util.JDBCExceptionReporter - Lock wait timeout exceeded; try restarting transaction
2018-03-20 05:08:26,886 [req.remoteHost=10.240.45.100, tenant=graxxx] [ajp--0_0_0_0_0_0_0_0-8009-23] ERROR o.h.e.d.AbstractFlushingEventListener - Could not synchronize database state with session
org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update
        at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA]
        [...]
        Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084) ~[mysql-connector-java-5.1.30.jar!/:na]
        [...]

[...]   

2018-03-20 05:08:32,886 [] [Transaction Reaper Worker 0] ERROR o.h.c.client.impl.ClientSessionImpl - Caught Exception
org.hornetq.api.core.HornetQException: Timed out waiting for response when sending packet 52
        at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:302) ~[hornetq-core-2.2.13.Final.jar!/:na]
        [...]
2018-03-20 05:08:32,887 [] [Transaction Reaper Worker 0] WARN com.arjuna.ats.arjuna - ARJUNA012091: Top-level abort of action 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
2018-03-20 05:08:32,887 [req.remoteHost=10.240.45.100, tenant=graxxx] [ajp--0_0_0_0_0_0_0_0-8009-23] WARN com.arjuna.ats.arjuna - ARJUNA012077: Abort called on already aborted atomic action 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5
2018-03-20 05:08:32,889 [] [Transaction Reaper Worker 0] WARN com.arjuna.ats.arjuna - ARJUNA012113: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] missed interrupt when cancelling TX 0:ffff0af02d68:-63fd0719:5ab027d7:5afb5 -- exiting as zombie (zombie count decremented to 0)

[...]

2018-03-20 05:08:36,407 [req.remoteHost=10.240.45.100, tenant=sngxxx] [ajp--0_0_0_0_0_0_0_0-8009-1] ERROR o.h.c.client.impl.ClientSessionImpl - XA operation failed Cannot start, session is already doing work in a transaction XidImpl (1278202944 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-77.75.0.0.0.7.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-81.-75.49 code:-6
2018-03-20 05:08:36,460 [req.remoteHost=10.240.45.100, tenant=sngxxx] [ajp--0_0_0_0_0_0_0_0-8009-1] ERROR o.h.ra.HornetQRASessionFactoryImpl - Could not create session
javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3c2cebaa[state=NORMAL managed connection=org.hornetq.ra.HornetQRAManagedConnection@903abac connection handles=0 lastUse=1521518916451 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@4ffbadd3 pool internal context=SemaphoreArrayListManagedConnectionPool@5ec1a591[pool=HornetQConnectionDefinition] xaResource=XAResourceWrapperImpl@14680555[xaResource=org.hornetq.ra.HornetQRAXAResource@2f78b5a pad=false overrideRmValue=false productName=HornetQ productVersion=2.0 jndiName=java:/Ms1Jms] txSync=null]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:604) ~[na:na]
        [...]/
        Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:474) ~[na:na]
        [...]
        Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff0af02d68:-63fd0719:5ab027d7:5b978 status: ActionStatus.ABORT_ONLY >
        [...]
        at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:552) ~[na:na]
        at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:282) ~[na:na]
        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:467) ~[na:na]
        ... 164 common frames omitted
2018-03-20 05:08:44,602 [req.remoteHost=10.240.45.100, tenant=sngxxx] [ajp--0_0_0_0_0_0_0_0-8009-25] ERROR o.h.c.client.impl.ClientSessionImpl - XA operation failed Cannot start, session is already doing work in a transaction XidImpl (1278202944 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-77.75.0.0.0.7.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-81.-75.49 code:-6
2018-03-20 05:08:44,663 [req.remoteHost=10.240.45.100, tenant=sngxxx] [ajp--0_0_0_0_0_0_0_0-8009-25] ERROR o.h.ra.HornetQRASessionFactoryImpl - Could not create session
javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3c2cebaa[state=NORMAL managed connection=org.hornetq.ra.HornetQRAManagedConnection@903abac connection handles=0 lastUse=1521518924643 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@4ffbadd3 pool internal context=SemaphoreArrayListManagedConnectionPool@5ec1a591[pool=HornetQConnectionDefinition] xaResource=XAResourceWrapperImpl@14680555[xaResource=org.hornetq.ra.HornetQRAXAResource@2f78b5a pad=false overrideRmValue=false productName=HornetQ productVersion=2.0 jndiName=java:/Ms1Jms] txSync=null]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:604) ~[na:na]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:467) ~[na:na]
[...]
2018-03-20 05:09:08,906 [req.remoteHost=10.240.45.100, tenant=sngxxx] [ajp--0_0_0_0_0_0_0_0-8009-6] ERROR o.h.c.client.impl.ClientSessionImpl - XA operation failed Cannot start, session is already doing work in a transaction XidImpl (1278202944 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-77.75.0.0.0.7.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-81.-75.49 code:-6
2018-03-20 05:09:08,957 [req.remoteHost=10.240.45.100, tenant=sngxxx] [ajp--0_0_0_0_0_0_0_0-8009-6] ERROR o.h.ra.HornetQRASessionFactoryImpl - Could not create session
javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3c2cebaa[state=NORMAL managed connection=org.hornetq.ra.HornetQRAManagedConnection@903abac connection handles=0 lastUse=1521518948948 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@4ffbadd3 pool internal context=SemaphoreArrayListManagedConnectionPool@5ec1a591[pool=HornetQConnectionDefinition] xaResource=XAResourceWrapperImpl@14680555[xaResource=org.hornetq.ra.HornetQRAXAResource@2f78b5a pad=false overrideRmValue=false productName=HornetQ productVersion=2.0 jndiName=java:/Ms1Jms] txSync=null]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:604) ~[na:na]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:467) ~[na:na]
[...]
2018-03-20 05:09:34,353 [req.remoteHost=10.240.45.100, tenant=graxxx] [ajp--0_0_0_0_0_0_0_0-8009-26] ERROR o.h.c.client.impl.ClientSessionImpl - XA operation failed Cannot start, session is already doing work in a transaction XidImpl (1278202944 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-77.75.0.0.0.7.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-81.-75.49 code:-6
2018-03-20 05:09:34,474 [req.remoteHost=10.240.45.100, tenant=graxxx] [ajp--0_0_0_0_0_0_0_0-8009-26] ERROR o.h.ra.HornetQRASessionFactoryImpl - Could not create session
javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3c2cebaa[state=NORMAL managed connection=org.hornetq.ra.HornetQRAManagedConnection@903abac connection handles=0 lastUse=1521518974452 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@4ffbadd3 pool internal context=SemaphoreArrayListManagedConnectionPool@5ec1a591[pool=HornetQConnectionDefinition] xaResource=XAResourceWrapperImpl@14680555[xaResource=org.hornetq.ra.HornetQRAXAResource@2f78b5a pad=false overrideRmValue=false productName=HornetQ productVersion=2.0 jndiName=java:/Ms1Jms] txSync=null]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:604) ~[na:na]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:467) ~[na:na]
[...]
2018-03-20 05:14:49,036 [tenant=bloxxx] [ajp--0_0_0_0_0_0_0_0-8009-14] ERROR o.h.c.client.impl.ClientSessionImpl - XA operation failed Cannot start, session is already doing work in a transaction XidImpl (1278202944 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-77.75.0.0.0.7.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.-16.45.104.-100.2.-8.-25.90.-80.39.-41.0.5.-81.-75.49 code:-6
2018-03-20 05:14:49,070 [tenant=bloxxx] [ajp--0_0_0_0_0_0_0_0-8009-14] ERROR o.h.ra.HornetQRASessionFactoryImpl - Could not create session
javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3c2cebaa[state=NORMAL managed connection=org.hornetq.ra.HornetQRAManagedConnection@903abac connection handles=0 lastUse=1521519289064 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@4ffbadd3 pool internal context=SemaphoreArrayListManagedConnectionPool@5ec1a591[pool=HornetQConnectionDefinition] xaResource=XAResourceWrapperImpl@14680555[xaResource=org.hornetq.ra.HornetQRAXAResource@2f78b5a pad=false overrideRmValue=false productName=HornetQ productVersion=2.0 jndiName=java:/Ms1Jms] txSync=null]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:604) ~[na:na]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:467) ~[na:na]
相关问题
最新问题