我正在运行4节点(4GB,4GB,4GB和6GB RAM)cassandra集群,并且有10个客户端应用程序,每个应用程序8个线程。
它正在落下
CassandraDaemon.java:185 - Exception in thread Thread[MessagingService-Incoming-/,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.nio.ByteBuffer.wrap(ByteBuffer.java:373) ~[na:1.8.0_77]
at java.nio.ByteBuffer.wrap(ByteBuffer.java:396) ~[na:1.8.0_77]
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:363) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:338) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.Row$RowSerializer.deserialize(Row.java:73) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.Row$RowSerializer.deserialize(Row.java:79) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.RangeSliceReply$RangeSliceReplySerializer.deserialize(RangeSliceReply.java:80) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.RangeSliceReply$RangeSliceReplySerializer.deserialize(RangeSliceReply.java:66) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) ~[apache-cassandra-2.2.5.jar:2.2.5]
INFO [ScheduledTasks:1] 2016-04-25 19:42:50,491 StatusLogger.java:56 - RequestResponseStage 0 0
在这个下降之前的日志中,我看到了乘法
DEBUG [SharedPool-Worker-7] 2016-04-25 19:38:16,830 AbstractQueryPager.java:89 - Got empty set of rows, considering pager exhausted
DEBUG [SharedPool-Worker-6] 2016-04-25 19:38:16,841 SliceQueryPager.java:92 - Querying next page of slice query; new filter: SliceQueryFilter [reversed=false, slices=[[, ]],
这也是所有日志中的另一个例外:
2016-04-27 19:43:22,675 CassandraDaemon.java:185 - Exception in thread Thread[CompactionExecutor:14,1,main]
java.lang.IllegalArgumentException: null
at java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_77]
at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:113) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:192) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:237) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:44) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:287) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:180) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:53) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:261) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:241) ~[apache-cassandra-2.2.5.jar:2.2.5]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:221) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:89) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:48) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.compaction.CompactionIterable.iterator(CompactionIterable.java:57) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:165) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:256) ~[apache-cassandra-2.2.5.jar:2.2.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_77]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
我试图取消注释 MAX_HEAP_SIZE =&#34; 4G&#34; HEAP_NEWSIZE =&#34; 800M&#34; 在cassandra-env.sh,
我尝试更改cassandra.yaml中的column_index_size_in_kb / rpc_max_threads,但仍然没有更改。此外,所有请求都非常慢。
在客户端,我使用
Cluster cluster = Cluster.builder()
.addContactPoints(/* 4 points */)
.build();
cluster.getConfiguration().getSocketOptions().setReadTimeoutMillis(100000);//because it always falls with read timeout
session = cluster.connect("keyspace");
mappingManager = new MappingManager(session);
如何才能解决我的计划瓶颈问题?我怎样才能阻止cassandra OOM?
UPD:在服务器上运行顶部,我得到了
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3597 root 20 0 9073600 4.536g 137728 S 100.4 58.2 24:24.25
我还将/etc/security/limits.d/cassandra.conf修改为最佳值