我从Cassandra获取数据时遇到了一些麻烦。
任务
我有一个初始加载的数据，其中包含810万个数据集，我将其保存在每行7700个数据集的块中。我们的想法是将每个分区保留在大约10万个列族中
在阅读了大约300个数据块或200万个数据集之后，应用程序变得缓慢而不是崩溃。
数据库需要在以后提供更多数据，所以我希望有人可以告诉我，出了什么问题。
注意
对于开发，cassandra安装在虚拟机上，ubuntu linux上有3CPU和13GB RAM以及OpenJDK8。
应用程序设置
数据库设置
java -ea -javaagent:./../lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms4G -Xmx4G -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+PerfDisableSharedMem -XX:CompileCommandFile=./../conf/hotspot_compiler -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=10000 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -Xloggc:./../logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Djava.net.preferIPv4Stack=true -Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC
数据库启动参数
手动设置Xms，Xmx和Xmn，启动脚本的计算值甚至更低。
int chunks = 1053; // chunk size is known
List<List<TransactionalData>> resultLists = new ArrayList<>(chunks);
for (int chunk = 1; chunk <= chunks; chunk++) {
Select select = QueryBuilder.select().from(TABLE_TRANSACTIONAL_DATA);
select.where(QueryBuilder.eq(CASSANDRA_COLUMN_JOB, jobId))
.and(QueryBuilder.eq(CASSANDRA_COLUMN_CLIENT, clientId))
.and(QueryBuilder.eq(CASSANDRA_COLUMN_RUN, run))
.and(QueryBuilder.eq(CASSANDRA_COLUMN_CHUNK, chunk));
StopWatch s = new StopWatch();
s.start();
List<TransactionalData> result = cassandraOperations.select(select, TransactionalData.class);
s.stop();
log.info("query [" + jobId + ", " + clientId + ", " + run + ", " + chunk + "] returned " + result.size() + " results in " + s.getTotalTimeMillis() + " ms");
resultLists.add(result);
}
申请代码
20:09:26.380 INFO ExportServiceImpl: query [initial, RM, 1, 1] returned 7700 results in 293 ms
[multiple reads between 300 - 400ms]
20:09:40.922 INFO ExportServiceImpl: query [initial, RM, 1, 36] returned 7700 results in 1160 ms
[multiple reads between 300 - 400ms]
20:09:51.785 INFO ExportServiceImpl: query [initial, RM, 1, 64] returned 7700 results in 1701 ms
[multiple reads between 300 - 400ms]
20:10:05.583 INFO ExportServiceImpl: query [initial, RM, 1, 97] returned 7700 results in 2476 ms
[multiple reads between 300 - 600ms]
20:11:00.956 INFO ExportServiceImpl: query [initial, RM, 1, 230] returned 7700 results in 3615 ms
[multiple reads between 300 - 3000ms and suddenly behaviour collapses]
20:11:17.707 INFO ExportServiceImpl: query [initial, RM, 1, 276] returned 7700 results in 370 ms
20:11:23.398 INFO ExportServiceImpl: query [initial, RM, 1, 277] returned 7700 results in 5691 ms
20:11:23.717 INFO ExportServiceImpl: query [initial, RM, 1, 278] returned 7700 results in 319 ms
20:11:24.033 INFO ExportServiceImpl: query [initial, RM, 1, 279] returned 7700 results in 316 ms
20:11:27.289 INFO ExportServiceImpl: query [initial, RM, 1, 280] returned 7700 results in 3256 ms
20:11:27.607 INFO ExportServiceImpl: query [initial, RM, 1, 281] returned 7700 results in 318 ms
20:11:30.380 INFO ExportServiceImpl: query [initial, RM, 1, 282] returned 7700 results in 2773 ms
20:11:30.669 INFO ExportServiceImpl: query [initial, RM, 1, 283] returned 7700 results in 288 ms
[multiple reads time increases up to 10000ms each read]
20:13:55.341 INFO ExportServiceImpl: query [initial, RM, 1, 323] returned 7700 results in 13813 ms
20:14:09.200 INFO ExportServiceImpl: query [initial, RM, 1, 324] returned 7700 results in 13859 ms
20:14:32.799 INFO ExportServiceImpl: query [initial, RM, 1, 325] returned 7700 results in 23599 ms
20:15:05.796 INFO ExportServiceImpl: query [initial, RM, 1, 326] returned 7700 results in 32997 ms
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:710)
at org.springframework.core.annotation.AnnotationUtils.synthesizeAnnotation(AnnotationUtils.java:1503)
at org.springframework.core.annotation.AnnotatedElementUtils.findMergedAnnotation(AnnotatedElementUtils.java:731)
at org.springframework.data.cassandra.mapping.BasicCassandraPersistentProperty.isCompositePrimaryKey(BasicCassandraPersistentProperty.java:109)
at org.springframework.data.cassandra.mapping.BasicCassandraPersistentProperty.getCompositePrimaryKeyType(BasicCassandraPersistentProperty.java:113)
at org.springframework.data.cassandra.mapping.BasicCassandraPersistentProperty.getCompositePrimaryKeyTypeInformation(BasicCassandraPersistentProperty.java:118)
at org.springframework.data.cassandra.mapping.BasicCassandraPersistentProperty.getCompositePrimaryKeyEntity(BasicCassandraPersistentProperty.java:394)
at org.springframework.data.cassandra.convert.MappingCassandraConverter.readPropertyFromRow(MappingCassandraConverter.java:164)
at org.springframework.data.cassandra.convert.MappingCassandraConverter$1.doWithPersistentProperty(MappingCassandraConverter.java:148)
at org.springframework.data.cassandra.convert.MappingCassandraConverter$1.doWithPersistentProperty(MappingCassandraConverter.java:143)
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:322)
at org.springframework.data.cassandra.convert.MappingCassandraConverter.readPropertiesFromRow(MappingCassandraConverter.java:143)
at org.springframework.data.cassandra.convert.MappingCassandraConverter.readEntityFromRow(MappingCassandraConverter.java:135)
at org.springframework.data.cassandra.convert.MappingCassandraConverter.readRow(MappingCassandraConverter.java:115)
at org.springframework.data.cassandra.convert.MappingCassandraConverter.read(MappingCassandraConverter.java:200)
at org.springframework.data.cassandra.core.CassandraConverterRowCallback.doWith(CassandraConverterRowCallback.java:47)
at org.springframework.data.cassandra.core.CassandraTemplate.select(CassandraTemplate.java:587)
at org.springframework.data.cassandra.core.CassandraTemplate.select(CassandraTemplate.java:354)
at AbstractExportService.exportData(AbstractExportService.java:79)
at ExportServiceImpl.exportData(ExportServiceImpl.java:33)
at ExportServiceImplTest.exportData(ExportServiceImplTest.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
申请日志
19:55:36,960 GCInspector.java:284 - ParNew GC in 715ms. CMS Old Gen: 2008935960 -> 2008954320; Par Eden Space: 671088640 -> 0; Par Survivor Space: 13605416 -> 16345128
19:55:36,967 StatusLogger.java:52 - Pool Name Active Pending Completed Blocked All Time Blocked
19:55:37,069 StatusLogger.java:56 - MutationStage 0 0 8321004 0 0
19:55:37,070 StatusLogger.java:56 - ReadStage 0 0 256399 0 0
19:55:37,071 StatusLogger.java:56 - RequestResponseStage 0 0 17 0 0
19:55:37,080 StatusLogger.java:56 - ReadRepairStage 0 0 0 0 0
19:55:37,081 StatusLogger.java:56 - CounterMutationStage 0 0 0 0 0
19:55:37,083 StatusLogger.java:56 - HintedHandoff 0 0 0 0 0
19:55:37,084 StatusLogger.java:56 - MiscStage 0 0 0 0 0
19:55:37,085 StatusLogger.java:56 - CompactionExecutor 0 0 61989 0 0
19:55:37,086 StatusLogger.java:56 - MemtableReclaimMemory 0 0 331 0 0
19:55:37,087 StatusLogger.java:56 - PendingRangeCalculator 0 0 1 0 0
19:55:37,088 StatusLogger.java:56 - GossipStage 0 0 0 0 0
19:55:37,089 StatusLogger.java:56 - MigrationStage 0 0 17 0 0
19:55:37,092 StatusLogger.java:56 - MemtablePostFlush 0 0 2387 0 0
19:55:37,095 StatusLogger.java:56 - ValidationExecutor 0 0 0 0 0
19:55:37,097 StatusLogger.java:56 - Sampler 0 0 0 0 0
19:55:37,099 StatusLogger.java:56 - MemtableFlushWriter 0 0 331 0 0
19:55:37,099 StatusLogger.java:56 - InternalResponseStage 0 0 0 0 0
19:55:37,100 StatusLogger.java:56 - AntiEntropyStage 0 0 0 0 0
19:55:37,102 StatusLogger.java:56 - CacheCleanupExecutor 0 0 0 0 0
19:55:37,102 StatusLogger.java:56 - Native-Transport-Requests 1 0 8356278 0 306061
19:55:37,116 StatusLogger.java:66 - CompactionManager 0 0
19:55:37,122 StatusLogger.java:78 - MessagingService n/a 0/0
19:55:37,122 StatusLogger.java:88 - Cache Type Size Capacity KeysToSave
19:55:37,126 StatusLogger.java:90 - KeyCache 50823767 104857600 all
19:55:37,126 StatusLogger.java:96 - RowCache 0 0 all
19:55:37,126 StatusLogger.java:103 - Table Memtable ops,data
19:55:37,144 StatusLogger.java:106 - system_distributed.parent_repair_history 0,0
19:55:37,144 StatusLogger.java:106 - system_distributed.repair_history 0,0
19:55:37,144 StatusLogger.java:106 - import_test.transactional_data 1199451,37053335
19:55:37,144 StatusLogger.java:106 - system.compaction_history 9,246
19:55:37,144 StatusLogger.java:106 - system.hints 0,0
19:55:37,144 StatusLogger.java:106 - system.schema_aggregates 0,0
19:55:37,144 StatusLogger.java:106 - system.IndexInfo 0,0
19:55:37,145 StatusLogger.java:106 - system.schema_columnfamilies 0,0
19:55:37,145 StatusLogger.java:106 - system.schema_triggers 0,0
19:55:37,145 StatusLogger.java:106 - system.schema_functions 0,0
19:55:37,145 StatusLogger.java:106 - system.size_estimates 520,3162
19:55:37,146 StatusLogger.java:106 - system.paxos 0,0
19:55:37,146 StatusLogger.java:106 - system.peer_events 0,0
19:55:37,146 StatusLogger.java:106 - system.range_xfers 0,0
19:55:37,146 StatusLogger.java:106 - system.compactions_in_progress 0,0
19:55:37,146 StatusLogger.java:106 - system.peers 0,0
19:55:37,146 StatusLogger.java:106 - system.schema_keyspaces 0,0
19:55:37,146 StatusLogger.java:106 - system.schema_usertypes 0,0
19:55:37,146 StatusLogger.java:106 - system.local 0,0
19:55:37,146 StatusLogger.java:106 - system.sstable_activity 418,1545
19:55:37,146 StatusLogger.java:106 - system.schema_columns 0,0
19:55:37,146 StatusLogger.java:106 - system.batchlog 0,0
19:55:37,147 StatusLogger.java:106 - system.available_ranges 0,0
19:55:37,147 StatusLogger.java:106 - system_auth.roles 0,0
19:55:37,147 StatusLogger.java:106 - system_auth.role_members 0,0
19:55:37,147 StatusLogger.java:106 - system_auth.resource_role_permissons_index 0,0
19:55:37,147 StatusLogger.java:106 - system_auth.role_permissions 0,0
19:55:37,147 StatusLogger.java:106 - system_traces.sessions 0,0
19:55:37,148 StatusLogger.java:106 - system_traces.events 0,0
20:14:32,062 Memtable.java:352 - Writing Memtable-sstable_activity@1527249591(1.509KiB serialized bytes, 616 ops, 0%/0% of on/off-heap limit)
20:14:32,062 Memtable.java:352 - Writing Memtable-size_estimates@907520469(3.088KiB serialized bytes, 780 ops, 0%/0% of on/off-heap limit)
20:14:32,128 Memtable.java:352 - Writing Memtable-compaction_history@173198051(0.240KiB serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
Cassandra系统日志
2017-01-19T20:11:10.127+0100: 117198.441: Total time for which application threads were stopped: 0.0020544 seconds, Stopping threads took: 0.0009831 seconds
{Heap before GC invocations=7026 (full 16):
par new generation total 737280K, used 664609K [0x00000006c0000000, 0x00000006f2000000, 0x00000006f2000000)
eden space 655360K, 100% used [0x00000006c0000000, 0x00000006e8000000, 0x00000006e8000000)
from space 81920K, 11% used [0x00000006ed000000, 0x00000006ed908710, 0x00000006f2000000)
to space 81920K, 0% used [0x00000006e8000000, 0x00000006e8000000, 0x00000006ed000000)
concurrent mark-sweep generation total 3375104K, used 1965475K [0x00000006f2000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 34481K, capacity 34783K, committed 35184K, reserved 1081344K
class space used 3678K, capacity 3772K, committed 3808K, reserved 1048576K
2017-01-19T20:11:12.390+0100: 117200.704: [GC (Allocation Failure) 2017-01-19T20:11:12.390+0100: 117200.704: [ParNew
Desired survivor size 41943040 bytes, new threshold 1 (max 1)
- age 1: 4114136 bytes, 4114136 total
: 664609K->7031K(737280K), 0.0122370 secs] 2630084K->1972513K(4112384K), 0.0123827 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
Heap after GC invocations=7027 (full 16):
par new generation total 737280K, used 7031K [0x00000006c0000000, 0x00000006f2000000, 0x00000006f2000000)
eden space 655360K, 0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e8000000)
from space 81920K, 8% used [0x00000006e8000000, 0x00000006e86ddfe8, 0x00000006ed000000)
to space 81920K, 0% used [0x00000006ed000000, 0x00000006ed000000, 0x00000006f2000000)
concurrent mark-sweep generation total 3375104K, used 1965481K [0x00000006f2000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 34481K, capacity 34783K, committed 35184K, reserved 1081344K
class space used 3678K, capacity 3772K, committed 3808K, reserved 1048576K
}
2017-01-19T20:11:17.399+0100: 117205.714: Total time for which application threads were stopped: 0.0274807 seconds, Stopping threads took: 0.0002973 seconds
{Heap before GC invocations=7028 (full 16):
par new generation total 737280K, used 670595K [0x00000006c0000000, 0x00000006f2000000, 0x00000006f2000000)
eden space 655360K, 100% used [0x00000006c0000000, 0x00000006e8000000, 0x00000006e8000000)
from space 81920K, 18% used [0x00000006ed000000, 0x00000006edee0c40, 0x00000006f2000000)
to space 81920K, 0% used [0x00000006e8000000, 0x00000006e8000000, 0x00000006ed000000)
concurrent mark-sweep generation total 3375104K, used 1965487K [0x00000006f2000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 34481K, capacity 34783K, committed 35184K, reserved 1081344K
class space used 3678K, capacity 3772K, committed 3808K, reserved 1048576K
2017-01-19T20:11:39.887+0100: 117228.201: [GC (Allocation Failure) 2017-01-19T20:11:39.887+0100: 117228.201: [ParNew
Desired survivor size 41943040 bytes, new threshold 1 (max 1)
- age 1: 4632832 bytes, 4632832 total
: 670595K->11530K(737280K), 0.0211463 secs] 2636082K->1977022K(4112384K), 0.0212837 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
Heap after GC invocations=7029 (full 16):
par new generation total 737280K, used 11530K [0x00000006c0000000, 0x00000006f2000000, 0x00000006f2000000)
eden space 655360K, 0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e8000000)
from space 81920K, 14% used [0x00000006e8000000, 0x00000006e8b428b0, 0x00000006ed000000)
to space 81920K, 0% used [0x00000006ed000000, 0x00000006ed000000, 0x00000006f2000000)
concurrent mark-sweep generation total 3375104K, used 1965491K [0x00000006f2000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 34481K, capacity 34783K, committed 35184K, reserved 1081344K
class space used 3678K, capacity 3772K, committed 3808K, reserved 1048576K
}
2017-01-19T20:12:10.131+0100: 117258.445: Total time for which application threads were stopped: 0.0005586 seconds, Stopping threads took: 0.0002172 seconds
{Heap before GC invocations=7030 (full 16):
par new generation total 737280K, used 664725K [0x00000006c0000000, 0x00000006f2000000, 0x00000006f2000000)
eden space 655360K, 100% used [0x00000006c0000000, 0x00000006e8000000, 0x00000006e8000000)
from space 81920K, 11% used [0x00000006ed000000, 0x00000006ed925448, 0x00000006f2000000)
to space 81920K, 0% used [0x00000006e8000000, 0x00000006e8000000, 0x00000006ed000000)
concurrent mark-sweep generation total 3375104K, used 1965495K [0x00000006f2000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 34481K, capacity 34783K, committed 35184K, reserved 1081344K
class space used 3678K, capacity 3772K, committed 3808K, reserved 1048576K
2017-01-19T20:12:42.762+0100: 117291.077: [GC (Allocation Failure) 2017-01-19T20:12:42.762+0100: 117291.077: [ParNew
Desired survivor size 41943040 bytes, new threshold 1 (max 1)
- age 1: 15961696 bytes, 15961696 total
: 664725K->19181K(737280K), 0.0197058 secs] 2630220K->1984679K(4112384K), 0.0198265 secs] [Times: user=0.05 sys=0.01, real=0.02 secs]
Heap after GC invocations=7031 (full 16):
par new generation total 737280K, used 19181K [0x00000006c0000000, 0x00000006f2000000, 0x00000006f2000000)
eden space 655360K, 0% used [0x00000006c0000000, 0x00000006c0000000, 0x00000006e8000000)
from space 81920K, 23% used [0x00000006e8000000, 0x00000006e92bb500, 0x00000006ed000000)
to space 81920K, 0% used [0x00000006ed000000, 0x00000006ed000000, 0x00000006f2000000)
concurrent mark-sweep generation total 3375104K, used 1965498K [0x00000006f2000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 34481K, capacity 34783K, committed 35184K, reserved 1081344K
class space used 3678K, capacity 3772K, committed 3808K, reserved 1048576K
}
2017-01-19T20:13:13.099+0100: 117321.413: Total time for which application threads were stopped: 0.0003673 seconds, Stopping threads took: 0.0000710 seconds
Cassandra gc log
20:14:03,669 SliceQueryPager.java:92 - Querying next page of slice query; new filter: SliceQueryFilter [reversed=false, slices=[[0010a7676a007a2949e490e58703550c222b0000047479706500, ]], count=5001, toGroup = 1]
20:14:03,670 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/import_test/transactional_data-79c13380dd9111e6b9e0d333906ebbeb/lb-49-big-Data.db
20:14:03,709 AbstractQueryPager.java:95 - Fetched 2701 live rows
20:14:03,709 AbstractQueryPager.java:112 - Got result (2701) smaller than page size (5001), considering pager exhausted
20:14:03,712 AbstractQueryPager.java:133 - Remaining rows to page: 2147475947
20:14:09,201 SliceQueryPager.java:92 - Querying next page of slice query; new filter: SliceQueryFilter [reversed=false, slices=[[, ]], count=5000, toGroup = 1]
20:14:09,202 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/import_test/transactional_data-79c13380dd9111e6b9e0d333906ebbeb/lb-49-big-Data.db
20:14:09,282 AbstractQueryPager.java:95 - Fetched 5000 live rows
20:14:09,282 AbstractQueryPager.java:133 - Remaining rows to page: 2147478647
20:14:10,153 ColumnFamilyStore.java:980 - forceFlush requested but everything is clean in batchlog
20:14:24,442 SliceQueryPager.java:92 - Querying next page of slice query; new filter: SliceQueryFilter [reversed=false, slices=[[0010a8ceba29bcec45b7b66323cbb506f09a0000047479706500, ]], count=5001, toGroup = 1]
20:14:24,443 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/import_test/transactional_data-79c13380dd9111e6b9e0d333906ebbeb/lb-49-big-Data.db
20:14:24,489 AbstractQueryPager.java:95 - Fetched 2701 live rows
20:14:24,489 AbstractQueryPager.java:112 - Got result (2701) smaller than page size (5001), considering pager exhausted
20:14:24,492 AbstractQueryPager.java:133 - Remaining rows to page: 2147475947
20:14:32,039 ColumnFamilyStore.java:927 - Enqueuing flush of sstable_activity: 13998 (0%) on-heap, 0 (0%) off-heap
20:14:32,050 ColumnFamilyStore.java:927 - Enqueuing flush of size_estimates: 42686 (0%) on-heap, 0 (0%) off-heap
20:14:32,064 ColumnFamilyStore.java:927 - Enqueuing flush of compaction_history: 1197 (0%) on-heap, 0 (0%) off-heap
20:14:32,062 Memtable.java:352 - Writing Memtable-size_estimates@907520469(3.088KiB serialized bytes, 780 ops, 0%/0% of on/off-heap limit)
20:14:32,086 Memtable.java:386 - Completed flushing /home/cassandra/data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/tmp-lb-198-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1484733263564, position=31840472)
20:14:32,128 Memtable.java:352 - Writing Memtable-compaction_history@173198051(0.240KiB serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
20:14:32,130 Memtable.java:386 - Completed flushing /home/cassandra/data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/tmp-lb-151-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1484733263564, position=31840472)
20:14:32,150 CompactionTask.java:142 - Compacting (81ce1850-de7b-11e6-b9e0-d333906ebbeb) [/home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/lb-198-big-Data.db:level=0, /home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/lb-199-big-Data.db:level=0, /home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/lb-200-big-Data.db:level=0, /home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/lb-197-big-Data.db:level=0, ]
20:14:32,190 FileCacheService.java:177 - Invalidating cache for /home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/tmplink-lb-201-big-Data.db
20:14:32,198 FileCacheService.java:177 - Invalidating cache for /home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/lb-200-big-Data.db
20:14:32,198 CompactionTask.java:224 - Compacted (81ce1850-de7b-11e6-b9e0-d333906ebbeb) 4 sstables to [/home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/lb-201-big,] to level=0. 4,012 bytes to 939 (~23% of original) in 40ms = 0.022388MB/s. 0 total partitions merged to 17. Partition merge counts were {1:12, 4:17, }
20:14:32,198 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/import_test/transactional_data-79c13380dd9111e6b9e0d333906ebbeb/lb-85-big-Data.db
20:14:32,201 FileCacheService.java:177 - Invalidating cache for /home/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/lb-197-big-Data.db
20:14:32,800 SliceQueryPager.java:92 - Querying next page of slice query; new filter: SliceQueryFilter [reversed=false, slices=[[, ]], count=5000, toGroup = 1]
20:14:32,800 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/import_test/transactional_data-79c13380dd9111e6b9e0d333906ebbeb/lb-49-big-Data.db
20:14:54,860 AbstractQueryPager.java:112 - Got result (2701) smaller than page size (5001), considering pager exhausted
20:14:54,862 AbstractQueryPager.java:133 - Remaining rows to page: 2147475947
20:15:05,797 SliceQueryPager.java:92 - Querying next page of slice query; new filter: SliceQueryFilter [reversed=false, slices=[[, ]], count=5000, toGroup = 1]
20:15:05,797 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/import_test/transactional_data-79c13380dd9111e6b9e0d333906ebbeb/lb-49-big-Data.db
20:15:05,876 AbstractQueryPager.java:95 - Fetched 5000 live rows
20:15:05,876 AbstractQueryPager.java:133 - Remaining rows to page: 2147478647
20:15:10,153 ColumnFamilyStore.java:980 - forceFlush requested but everything is clean in batchlog
20:15:51,040 SliceQueryPager.java:92 - Querying next page of slice query; new filter: SliceQueryFilter [reversed=false, slices=[[0010a5a53060d5d043f2832958356689c4ee0000047479706500, ]], count=5001, toGroup = 1]
20:15:51,040 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/import_test/transactional_data-79c13380dd9111e6b9e0d333906ebbeb/lb-49-big-Data.db
20:15:51,083 AbstractQueryPager.java:95 - Fetched 2701 live rows
20:15:51,083 AbstractQueryPager.java:112 - Got result (2701) smaller than page size (5001), considering pager exhausted
20:15:51,086 AbstractQueryPager.java:133 - Remaining rows to page: 2147475947
20:18:10,161 ColumnFamilyStore.java:980 - forceFlush requested but everything is clean in batchlog
20:18:42,277 FileCacheService.java:102 - Evicting cold readers for /home/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/lb-134-big-Data.db
Cassandra调试日志
Recipe