Question

我对Cassandra的自动压缩有一个奇怪的问题。我在Debian 8系统上使用cassandra 3.7，并将大约70GB的数据推送到cassandra节点（它是一个RF = 1的单个节点用于测试目的），运行 来自命令行的nodetool compactionstats ：

root@cassandra01:~# nodetool compactionstats
pending tasks: 280
- system.batches: 280

并且不显示其他信息。检查 system.log 我明白了：

ERROR [CompactionExecutor:74] 2016-08-23 19:41:30,006 CassandraDaemon.java:217 - Exception in thread Thread[CompactionExecutor:74,1,main]
java.lang.AssertionError: null
        at org.apache.cassandra.io.compress.CompressionMetadata$Chunk.<init>(CompressionMetadata.java:475) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:240) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:158) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:99) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:44) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:135) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:181) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:192) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:271) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:182) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:141) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:187) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.7.jar:3.7]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-3.7.jar:3.7]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
ERROR [Reference-Reaper:1] 2016-08-23 19:42:05,511 Ref.java:203 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@57d68945) to class org.apache.cassandra.io.util.SegmentedFile$Cleanup@831676520:/cassandra/disk1/system/batches-919a4bc57a333573b03e13fc3f68b465/mb-44056-big-Index.db was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2016-08-23 19:42:05,511 Ref.java:203 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@39229a12) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1862266673:Memory@[7fb261a66020..7fb261a69220) was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2016-08-23 19:42:05,511 Ref.java:203 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@d80df0a) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@791904242:[Memory@[0..188), Memory@[0..f50)] was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2016-08-23 19:42:05,523 Ref.java:203 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@26c920c) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@326857495:/cassandra/disk1/system/batches-919a4bc57a333573b03e13fc3f68b465/mb-44056-big-Data.db was not released before the reference was garbage collected

在我看来，cassandra挂在CompressionMetadata类中的某个位置并获得异常，然后稍后 Reference-Reaper 找到一个内存因未捕获的异常而泄漏然而，问题永远不会消失，因为这些错误每隔30秒就会在日志中显示出来。

以前有人见过那件事吗？

感谢。

Answer 1

之前我还没有看到过这个错误，但听起来好像你把数据推得太快了。它似乎能够处理写入负载，但可能在一段时间后它无法跟上压缩并继续创建更多的压缩作业，直到出现溢出。可能你现在有一些损坏的SSTable。

尝试以较慢的速率推送数据并在推送期间监视压缩操作，以确保压缩作业能够及时完成而不会累积。如果您不能降低写入速率，那么您可能需要更多节点来分割负载。

我见过的最接近的错误是在推送大量数据后，compactionstats中显示了几个排队的压缩作业，但是没有一个会启动，他们只是坐在那里。我能够通过滚动重新启动集群中的所有节点来清除它。

坚持压实

1 个答案: