Question

我正在运行一个JBoss 7 ha集群，在2台服务器上有4个节点。几个星期后，偶尔抛出以下异常，有时每天2或3次，然后每周1次，每次2次。当有更多用户活跃时，看起来抛出异常的可能性更高。

 21:38:24,337 ERROR [org.infinispan.remoting.ReplicationQueue] (Scheduled-replicationQueue-thread-0) [, , ] ISPN000072: Failed replicating 3 elements in replication queueorg.infinispan.CacheException: java.lang.OutOfMemoryError: Java heap space
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:166) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:181) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.ReplicationQueueImpl.flush(ReplicationQueueImpl.java:159) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.ReplicationQueueImpl$1.run(ReplicationQueueImpl.java:112) [infinispan-core-5.1.0.BETA5.jar:]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_29]
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [:1.6.0_29]
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [:1.6.0_29]
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [:1.6.0_29]
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_29]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_29]
      at java.lang.Thread.run(Thread.java:662) [:1.6.0_29] Caused by: java.lang.OutOfMemoryError: Java heap space
      at org.infinispan.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:104) [infinispan-api-5.1.0.BETA5.jar:]
      at java.io.FilterOutputStream.write(FilterOutputStream.java:60) [:1.6.0_29]
      at java.io.FilterOutputStream.write(FilterOutputStream.java:108) [:1.6.0_29]
      at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:83) [jboss-marshalling-1.3.4.GA.jar:]
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:113) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:66) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:389) [infinispan-core-5.1.0.BETA5.jar:]
      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
      at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) [jboss-marshalling-1.3.4.GA.jar:]
      at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) [jboss-marshalling-1.3.4.GA.jar:]
      at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:59) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:91) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:57) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:84) [infinispan-commons-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.marshallCall(CommandAwareRpcDispatcher.java:211) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:232) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:110) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:411) [infinispan-core-5.1.0.BETA5.jar:]
      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:153) [infinispan-core-5.1.0.BETA5.jar:]
      ... 12 more

我已经尝试将host.xml中的JVM堆和permgen大小增加到8/16 GB而没有成功。

 <jvms>
      <jvm name="default">
           <heap size="16g" max-size="16g"/>
           <permgen size="8g" max-size="8g"/>
      </jvm>
 </jvms>

有什么想法吗？或者是否可以增加Infispan堆大小？

Answer 1

存在内存泄漏（最有可能）或发生大量调用。

在内存泄漏的情况下，Stacktrace可能与内存泄漏无关。它显示了什么是打破骆驼背部的吸管。

添加“-XX：+ HeapDumpOnOutOfMemoryError”的JVM arg来获取堆转储可能有助于找到真正的原因。

此外，Perm Gen可以从8gb无害地减少到几百MB，这与系统遇到的内存不足错误无关。

Answer 2

这个主题已经有几个月了，但我能够自己解决这个问题。

该问题与将数据存储在用户会话中的特定应用程序有关。在一些特殊和非常罕见的情况下，数据量仍然太多，序列化和反序列化的时间也增加了。例如，普通用户会话添加了10个自定义属性，导致问题的会话添加了超过10000个属性。

解决方案（在我的情况下！）不再将此属性存储在用户的会话中，而是将它们添加到分布式ehcache中。

此工具对于发现问题非常有帮助：http://messadmin.sourceforge.net/

Infinispan：java.lang.OutOfMemoryError：Java堆空间

2 个答案: