cassandra 2.1线程cpu用户时间100

时间:2014-05-09 09:10:29

标签: java cassandra

我使用几个进程通过tcp编写cassandra。一开始很好。内存达到50%后,我的进程有很多超时。此时一个cassandra线程的cpu用户时间很长时间达到100%,而system.log如下:

INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,996 StatusLogger.java:115 - RRD2.AVE_WK                         1906,2098
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - RRD2.RRA                                  0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.schema_triggers                    0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.local                              0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.peers                              0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,997 StatusLogger.java:115 - system.batchlog                           0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.NodeIdInfo                         0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.compactions_in_progress                 0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.schema_usertypes                   0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.schema_keyspaces                   0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.compaction_history                 0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,998 StatusLogger.java:115 - system.paxos                              0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.schema_columns                     0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.schema_columnfamilies                 0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.IndexInfo                          0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.range_xfers                        0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:43,999 StatusLogger.java:115 - system.peer_events                        0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system.hints                              0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system.sstable_activity                   0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system_traces.sessions                    0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:01:44,000 StatusLogger.java:115 - system_traces.events                      0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:06,068 GCInspector.java:116 - GC for ConcurrentMarkSweep: 21284 ms for 1 collections, 4226524688 used; max is 4242538496
INFO  [ScheduledTasks:1] 2014-05-09 15:02:06,069 StatusLogger.java:51 - Pool Name                    Active   Pending      Completed   Blocked  All Time Blocked
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,897 StatusLogger.java:66 - CounterMutationStage              0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,898 StatusLogger.java:66 - ReadStage                         0         0            106         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,898 StatusLogger.java:66 - RequestResponseStage              0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,899 StatusLogger.java:66 - ReadRepairStage                   0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,899 StatusLogger.java:66 - MutationStage                    32        92          20620         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,900 StatusLogger.java:66 - GossipStage                       0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,900 StatusLogger.java:66 - AntiEntropyStage                  0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,901 StatusLogger.java:66 - MigrationStage                    0         0              8         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,901 StatusLogger.java:66 - MiscStage                         0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,902 StatusLogger.java:66 - MemtableFlushWriter               2         8             23         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,902 StatusLogger.java:66 - PendingRangeCalculator            0         0              1         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,903 StatusLogger.java:66 - MemtablePostFlush                 1        46             69         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,903 StatusLogger.java:66 - commitlog_archiver                0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,904 StatusLogger.java:66 - InternalResponseStage             0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,904 StatusLogger.java:66 - HintedHandoff                     0         0              0         0                 0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:75 - CompactionManager                 0         0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:87 - MessagingService                n/a       0/0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:97 - Cache Type                     Size                 Capacity               KeysToSave
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:99 - KeyCache                       2208                104857600                      all
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,905 StatusLogger.java:105 - RowCache                          0                        0                      all
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:112 - ColumnFamily                Memtable ops,data
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.DS                                   0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.CDP                       2226732,476355
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.AVE_MO                         3040,3722
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.PDP                        923845,604642
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,906 StatusLogger.java:115 - RRD2.AVE_HR                   943064,83811099
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,907 StatusLogger.java:115 - RRD2.AVE_YR                           120,164
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,907 StatusLogger.java:115 - RRD2.AVE_WK                         3040,3722
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,907 StatusLogger.java:115 - RRD2.RRA                                  0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,916 StatusLogger.java:115 - system.schema_triggers                    0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,916 StatusLogger.java:115 - system.local                              0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,916 StatusLogger.java:115 - system.peers                              0,0
INFO  [ScheduledTasks:1] 2014-05-09 15:02:21,936 StatusLogger.java:115 - system.batchlog                           0,0
WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2014-05-09 15:08:56,334 AbstractCommitLogService.java:105 - Out of 11 commit log syncs over the past 321s with average duration of 26478.00ms, 7 have exceeded the configured commit interval by an average of 31464.57ms

任何人都知道发生了什么?

2 个答案:

答案 0 :(得分:1)

这一行表示您正在重载节点,使其运行java堆空间:

INFO  [ScheduledTasks:1] 2014-05-09 15:02:06,068 GCInspector.java:116 - GC for ConcurrentMarkSweep: 21284 ms for 1 collections, 4226524688 used; max is 4242538496

您可能需要调回一些缓存或记事本或其他内容。如果你进行堆转储,你可以弄清楚是什么在使用所有内存。

答案 1 :(得分:0)

仅是关于为何看到StatusLogger条目的后续操作。状态记录器会在Cassandra出现“压力”的特定时间转储一些关键指标。正如Zanson指出的那样,由于运行垃圾回收,您特别看到了这一点。

默认情况下,有一个名为gc_log_threshold_in_ms的配置项,默认为200ms(可在配置中调整),任何大于200ms的垃圾回收都会记录您所看到的状态记录器项。

此外,还有一个名为gc_warn_threshhold_in_ms的配置条目,如果垃圾收集运行的时间大于1000ms,它将以WARN日志级别而不是INFO记录这些条目(默认情况下,可调)。