Question

从最近几天我看到cassandra集群中的一些节点由于下面的错误而DSE正在关闭，我需要终止Java进程并重新启动DSE服务。

我已经交叉检查读取和写入并且压缩没有任何看起来可疑，但我发现在问题发生之前这些服务器上的完整Gc暂停。这可能是什么原因如何调试？如何在这些特定节点上细化为什么当这些节点提供与簇的其余部分相同的请求时会发生这种情况？

是否因为Full Gc没有正常执行而发生这种情况，我们使用G1GC和DSE 4.8.3

    ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100  JVMStabilityInspector.java:117 - JVM state determined to be unstable.  Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74]
at org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74]
            at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

    ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100  SEPWorker.java:141 - Failed to execute task, unexpected exception killed worker: {}
    java.lang.IllegalStateException: Shutdown in progress
            at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74]
            at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_74]
            at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

    INFO  [Thread-6] 2016-12-27 10:14:56,150  DseDaemon.java:420 - DSE shutting down...

Answer 1

检查Cassandra表中是否存在任何大分区，以及是否有任何查询正在阅读很多墓碑。

JVM状态确定为不稳定。强行退出。什么是Java Stability Inspector？为什么它会停止DSE？

1 个答案: