由于下面的错误，Cassandra集群正在关闭，我需要终止Java进程并重新启动DSE服务。
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
WARN [ReadStage-16] 2017-11-24 02:51:36,899
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
thread Thread[ReadStage-16,5,main]: {}
java.lang.OutOfMemoryError: Java heap space
我们是一个4节点集群
我正在使用G1垃圾收集器而不是CMS收集器
1
MAX_HEAP_SIZE="32G"
HEAP_NEWSIZE="8000M"
OS: CentOS - 7 RAM: 142 GB Swap: 3 GB Processor: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz Core: 40 Core Disk: 2.5T
2
MAX_HEAP_SIZE="32G"
HEAP_NEWSIZE="8000M"
OS: CentOS - 7 RAM: 125 GB Swap: 3 GB Processor: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz Core: 40 Core Disk: 2.2T
3
MAX_HEAP_SIZE="32G"
HEAP_NEWSIZE="8000M"
OS: CentOS - 7 RAM: 125 GB Swap: 3 GB Processor: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz Core: 40 Core Disk: 2 TB
4
MAX_HEAP_SIZE="32G"
HEAP_NEWSIZE="8000M"
OS: CentOS - 7 RAM: 125 GB Swap: 3 GB Processor: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Core: 12 Core Disk: 2.7 TB
我经常在我的debug.log中得到这个
DEBUG [GossipStage:1] FailureDetector.java:456 - Ignoring interval time of 2019820650 for /x.x.x.x
我已经完成了StackOverFlow中提出的一些问题，但没有得到任何正确答案。
以下是错误日志。
INFO [ReadStage-8] 2017-11-20 10:06:14,947 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
WARN [PERIODIC-COMMIT-LOG-SYNCER] 2017-11-20 10:09:03,480 NoSpamLogger.java:94 - Out of 19 commit log syncs over the past 177s with average duration of 628.42ms, 1 have exceeded the configured commit interval by an average of 1914.00ms
INFO [ReadStage-3] 2017-11-20 10:09:23,305 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
ERROR [Native-Transport-Requests-119] 2017-11-19 01:30:47,005 JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at ch.qos.logback.core.pattern.PatternLayoutBase.writeLoopOnConverters(PatternLayoutBase.java:116) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:149) ~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:39) ~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.core.encoder.LayoutWrappingEncoder.doEncode(LayoutWrappingEncoder.java:134) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.core.OutputStreamAppender.writeOut(OutputStreamAppender.java:194) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.core.FileAppender.writeOut(FileAppender.java:220) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:219) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:182) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48) ~[logback-core-1.1.3.jar:na]
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:273) ~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:260) ~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:442) ~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396) ~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.classic.Logger.error(Logger.java:559) ~[logback-classic-1.1.3.jar:na]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:128) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [apache-cassandra-3.9.0.jar:3.9.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
ERROR [Native-Transport-Requests-68] 2017-11-19 01:30:47,008 Message.java:617 - Unexpected exception during request; channel = [id: 0xf0c5b676, L:0.0.0.0/0.0.0.0:9042 ! R:/209.58.139.155:54814]
java.lang.OutOfMemoryError: Java heap space
at org.mindrot.jbcrypt.BCrypt.key(BCrypt.java:548) ~[jbcrypt-0.3m.jar:na]
at org.mindrot.jbcrypt.BCrypt.crypt_raw(BCrypt.java:628) ~[jbcrypt-0.3m.jar:na]
at org.mindrot.jbcrypt.BCrypt.hashpw(BCrypt.java:692) ~[jbcrypt-0.3m.jar:na]
at org.mindrot.jbcrypt.BCrypt.checkpw(BCrypt.java:763) ~[jbcrypt-0.3m.jar:na]
at org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:89) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.auth.PasswordAuthenticator.access$300(PasswordAuthenticator.java:59) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.auth.PasswordAuthenticator$PlainTextSaslAuthenticator.getAuthenticatedUser(PasswordAuthenticator.java:220) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [apache-cassandra-3.9.0.jar:3.9.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
ERROR [ReadStage-9] 2017-11-19 01:30:47,008 JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.db.Slice.extractValues(Slice.java:114) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Slice.make(Slice.java:97) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.filter.ClusteringIndexNamesFilter.getSlices(ClusteringIndexNamesFilter.java:133) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.initializeIterator(UnfilteredRowIteratorWithLowerBound.java:105) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:99) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:114) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:46) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:500) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:360) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:666) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:615) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:492) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.index.internal.composites.CompositesSearcher$1.prepareNext(CompositesSearcher.java:169) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.index.internal.composites.CompositesSearcher$1.hasNext(CompositesSearcher.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:289) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:145) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:138) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:134) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:323) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_144]