我一直在遇到3节点,RF 3,Cassandra 2.1.16群集的问题,并会尽可能准确地描述它们,考虑到我现在所掌握的知识。
向表中添加新行:
a)在我执行nodetool修复或
之前不会复制到节点3b)它将行复制到节点3,但留下一些列为null或过时的列。
可能不相关但这似乎发生在您首次插入的行中,然后执行读写操作,而不是仅在附加表中。
我知道Cassandra并不想像another question中所解释的那样重复使用列名或模式名称,并且我调整了我的传输管道以创建新命名的结构,也就是那里似乎是数据和模式之间松散耦合的关系 - 有时候模式与数据不匹配。
我在现有表格中添加了一些新的表格和列,现在我只在节点3上每隔几分钟就会出现这种情况,这显然与压缩有关:
INFO [CompactionExecutor:48525] 2017-03-27 11:08:04,851 CompactionTask.java:141 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7719-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7722-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7721-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7720-Data.db')]
INFO [CompactionExecutor:48524] 2017-03-27 11:08:04,851 CompactionTask.java:141 - Compacting [SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-3-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-1-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-2-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-5-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-4-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-462-Data.db')]
INFO [CompactionExecutor:48525] 2017-03-27 11:08:04,866 CompactionTask.java:274 - Compacted 4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7723,]. 476 bytes to 185 (~38% of original) in 14ms = 0.012602MB/s. 3 total partitions merged to 1. Partition merge counts were {1:1, 2:1, }
INFO [CompactionExecutor:48524] 2017-03-27 11:08:04,934 ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 148 (0%) on-heap, 20 (0%) off-heap
INFO [MemtableFlushWriter:6333] 2017-03-27 11:08:04,934 Memtable.java:347 - Writing Memtable-compactions_in_progress@1966348368(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:6333] 2017-03-27 11:08:04,935 Memtable.java:382 - Completed flushing /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-7724-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1485986982740, position=11004802)
ERROR [CompactionExecutor:48524] 2017-03-27 11:08:04,954 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:48524,1,main]
java.lang.RuntimeException: 6865617274707265736372697074696f6e73 is not defined as a collection
at org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.composites.CompoundSparseCellNameType$WithCollection.compare(CompoundSparseCellNameType.java:287) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.composites.AbstractCellNameType$4.compare(AbstractCellNameType.java:82) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.composites.AbstractCellNameType$4.compare(AbstractCellNameType.java:79) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.utils.MergeIterator$Candidate.compareTo(MergeIterator.java:154) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.utils.MergeIterator$Candidate.compareTo(MergeIterator.java:131) ~[apache-cassandra-2.1.16.jar:2.1.16]
at java.util.PriorityQueue.siftUpComparable(PriorityQueue.java:656) ~[na:1.8.0_121]
at java.util.PriorityQueue.siftUp(PriorityQueue.java:647) ~[na:1.8.0_121]
at java.util.PriorityQueue.offer(PriorityQueue.java:344) ~[na:1.8.0_121]
at java.util.PriorityQueue.add(PriorityQueue.java:321) ~[na:1.8.0_121]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) ~[apache-cassandra-2.1.16.jar:2.1.16]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:174) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:193) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-2.1.16.jar:2.1.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
我倾倒了一个"描述键空间KEYSPACENAME;"从每个节点到一个文件并做了一个差异,所有3个文件是相同的。还在每个节点上进行了nodetool修复。
另一方面,在我对表进行更改后,我发现了一些大量的时钟漂移(4-15秒),然后通过将节点放置为ntp对等体来修复ntp服务配置。
答案 0 :(得分:1)
架构操作不使用常规写入路径。它们通过单独的机制进行复制。
使用nodetool describelcuster
进行检查,您可能会看到每个节点都使用不同的架构版本。
通常的解决方案是滚动重启。