Question

我们有一个运行ehcache多年的生产应用程序 - 目前是ehcache 2.10.1。最近我们注意到运行ehcache的所有服务器上的死锁 - 大约每天10个 - 如下所示。

也许我们的负载随着时间的推移而增加，并且太过努力推动ehcache。在死锁几个小时后，此错误一直在创建JVM挂起状态。有任何想法吗？

 /home>  ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args|grep 89366 | awk ' $5 >= 2 ' (Wildfly PID 89366, find Linux lightweight process (LWP) with CPU usage greater than 2%) 
 89366  81979  118 prddepl+ 98.2 09:45    03:57:44 /apps/jdk1.8.0_92/bin/java -server -Xms1024m -Xmx5120m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/apps/wildfly/standalone/log -Dsun.net.inetaddr.ttl=60 -Djava.awt.headless=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.tanukisoftware.wrapper.WrapperManager.mbean=false -Dlogging.configuration=file:/apps/wildfly/standalone/configuration/logging.properties -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dorg.jboss.logging.Logger.pluginClass=org.jboss.logging.logmanager.LoggerPluginImpl -Djava.io.tmpdir=/tmp -Djava.net.preferIPv4Stack=true -javaagent:/apps/wildfly/standalone/lib/notsoserial-1.0-SNAPSHOT.jar -Dnotsoserial.whitelist=/apps/wildfly/bin/is-deserialized.txt -XX:-OmitStackTraceInFastThrow -Djava.library.path=/apps/wildfly/standalone/lib -classpath /apps/wildfly/standalone/lib/wrapper.jar:/apps/wildfly/standalone/lib/notsoserial-1.0-SNAPSHOT.jar:/apps/wildfly/standalone/lib/wrapper.jar -Dwrapper.key=u02jJraVjSs0PwNd -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=89364 -Dwrapper.version=3.5.13 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperJarApp /apps/wildfly/jboss-modules.jar -mp /apps/wildfly/modules -logmodule org.jboss.logmanager -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/apps/wildfly -b=0.0.0.0

[my-web1.idm.local(myuser)] 
 /home> printf "%x\n" 81979 (convert LWP to hex) 
1403b

[my-web1.idm.local(myuser)] 
 /home> jstack -l  89366 | grep -A 10 1403b
"EventsThread <3096>" #248148 prio=5 os_prio=0 tid=0x00007f815c076000 nid=0x1403b runnable [0x00007f81326e5000]
   java.lang.Thread.State: RUNNABLE
        at net.sf.ehcache.store.disk.Segment.clearFaultedBit(Segment.java:1051)
        at net.sf.ehcache.store.disk.DiskStore.clearFaultedBit(DiskStore.java:675)
        at net.sf.ehcache.store.CacheStore.flush(CacheStore.java:436)
        at net.sf.ehcache.Cache.flush(Cache.java:2650)
        - locked <0x000000069dd8d428> (a net.sf.ehcache.Cache)
        at com.mycompany.ticker.cache.ExternalTickerCacheManager.onUpdate(ExternalTickerCacheManager.java:945)

编辑：请参阅下面的ehcache配置。

<ehcache name="mycompany-tickerEhcache-dev" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="ehcahce.xsd">
     <!--Default Cache configuration. These will applied to caches programmatically created through
    the CacheManager.

    The following attributes are required for defaultCache:

    maxInMemory       - Sets the maximum number of objects that will be created in memory
    eternal           - Sets whether elements are eternal. If eternal,  timeouts are ignored and the element
                        is never expired.
    timeToIdleSeconds - Sets the time to idle for an element before it expires.
                        i.e. The maximum amount of time between accesses before an element expires
                        Is only used if the element is not eternal.
                        Optional attribute. A value of 0 means that an Element can idle for infinity
    timeToLiveSeconds - Sets the time to live for an element before it expires.
                        i.e. The maximum time between creation time and when an element expires.
                        Is only used if the element is not eternal.
    overflowToDisk    - Sets whether elements can overflow to disk when the in-memory cache
                        has reached the maxInMemory limit.
    timeToIdleSeconds="500"
    timeToLiveSeconds="700"
    -->

<diskStore path="/apps/wildfly/conf/tickerEhcache-dev"/>

<defaultCache maxElementsInMemory="0" eternal="true" memoryStoreEvictionPolicy="LRU" diskPersistent="true" >
</defaultCache>

<cache name="com.mycompany.ticker.ehcache"
  maxElementsInMemory="0"
  eternal="true"
  overflowToDisk="false"
  memoryStoreEvictionPolicy="LFU"
  diskPersistent="true" >

  <bootstrapCacheLoaderFactory class="com.mycompany.ticker.cache.MyCompanyDiskStoreBootstrapCacheLoaderFactory"
                               properties="bootstrapAsynchronously=true, interval=5"/>
</cache>

<cache name="com.mycompany.ticker.history.ehcache" eternal="false" maxElementsInMemory="0" overflowToDisk="false" diskPersistent="true" >

<cacheWriter writeMode="write_behind" notifyListenersOnException="true" maxWriteDelay="0" 
                   writeCoalescing="false" writeBatching="true" writeBatchSize="25" 
                   retryAttempts="1" retryAttemptDelaySeconds="1" rateLimitPerSecond="0">
</cacheWriter>
</cache>
</ehcache>

Ehcache线程死锁

0 个答案: