我使用几个进程通过tcp编写cassandra。一开始很好。内存达到50%后,我的进程有很多超时。此时一个cassandra线程的cpu用户时间很长时间达到100%,而system.log如下:
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,996 StatusLogger.java:115 - RRD2.AVE_WK 1906,2098
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - RRD2.RRA 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.local 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.peers 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.batchlog 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.compactions_in_progress 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.schema_usertypes 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.compaction_history 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.paxos 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.schema_columns 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.range_xfers 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.peer_events 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system.hints 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system.sstable_activity 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system_traces.events 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:02:06,068 GCInspector.java:116 - GC for ConcurrentMarkSweep: 21284 ms for 1 collections, 4226524688 used; max is 4242538496
INFO [ScheduledTasks:1] 2014-05-09 15:02:06,069 StatusLogger.java:51 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,897 StatusLogger.java:66 - CounterMutationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,898 StatusLogger.java:66 - ReadStage 0 0 106 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,898 StatusLogger.java:66 - RequestResponseStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,899 StatusLogger.java:66 - ReadRepairStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,899 StatusLogger.java:66 - MutationStage 32 92 20620 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,900 StatusLogger.java:66 - GossipStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,900 StatusLogger.java:66 - AntiEntropyStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,901 StatusLogger.java:66 - MigrationStage 0 0 8 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,901 StatusLogger.java:66 - MiscStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,902 StatusLogger.java:66 - MemtableFlushWriter 2 8 23 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,902 StatusLogger.java:66 - PendingRangeCalculator 0 0 1 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,903 StatusLogger.java:66 - MemtablePostFlush 1 46 69 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,903 StatusLogger.java:66 - commitlog_archiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,904 StatusLogger.java:66 - InternalResponseStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,904 StatusLogger.java:66 - HintedHandoff 0 0 0 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:75 - CompactionManager 0 0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:87 - MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:97 - Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:99 - KeyCache 2208 104857600 all
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:105 - RowCache 0 0 all
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:112 - ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.DS 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.CDP 2226732,476355
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.AVE_MO 3040,3722
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.PDP 923845,604642
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.AVE_HR 943064,83811099
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,907 StatusLogger.java:115 - RRD2.AVE_YR 120,164
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,907 StatusLogger.java:115 - RRD2.AVE_WK 3040,3722
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,907 StatusLogger.java:115 - RRD2.RRA 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,916 StatusLogger.java:115 - system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,916 StatusLogger.java:115 - system.local 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,916 StatusLogger.java:115 - system.peers 0,0
INFO [ScheduledTasks:1] 2014-05-09 15:02:21,936 StatusLogger.java:115 - system.batchlog 0,0
WARN [PERIODIC-COMMIT-LOG-SYNCER] 2014-05-09 15:08:56,334 AbstractCommitLogService.java:105 - Out of 11 commit log syncs over the past 321s with average duration of 26478.00ms, 7 have exceeded the configured commit interval by an average of 31464.57ms
任何人都知道发生了什么?
答案 0 :(得分:1)
这一行表示您正在重载节点,使其运行java堆空间:
INFO [ScheduledTasks:1] 2014-05-09 15:02:06,068 GCInspector.java:116 - GC for ConcurrentMarkSweep: 21284 ms for 1 collections, 4226524688 used; max is 4242538496
您可能需要调回一些缓存或记事本或其他内容。如果你进行堆转储,你可以弄清楚是什么在使用所有内存。
答案 1 :(得分:0)
仅是关于为何看到StatusLogger条目的后续操作。状态记录器会在Cassandra出现“压力”的特定时间转储一些关键指标。正如Zanson指出的那样,由于运行垃圾回收,您特别看到了这一点。
默认情况下,有一个名为gc_log_threshold_in_ms
的配置项,默认为200ms(可在配置中调整),任何大于200ms的垃圾回收都会记录您所看到的状态记录器项。
此外,还有一个名为gc_warn_threshhold_in_ms
的配置条目,如果垃圾收集运行的时间大于1000ms,它将以WARN
日志级别而不是INFO
记录这些条目(默认情况下,可调)。