卡桑德拉高cpu

时间:2018-05-14 14:24:55

标签: linux cassandra garbage-collection cpu

我有150个节点的集群,Cassandra 2.2.x,8个cpu和64 GB ram。 我的节点偶尔有100%的cpu,但其余的节点看起来都很好。

当我有100%cpu:

我的tpstats

没有任何压缩待决。

INFO  [Service Thread] 2018-05-12 18:03:51,636 GCInspector.java:284 - G1 Young Generation GC in 262ms.  G1 Eden Space: 1291845632 -> 0; G1 Old Gen: 10237756144 -> 7451103472; G1 Survivor Space: 419430400 -> 134217728;
INFO  [Service Thread] 2018-05-12 18:03:54,130 GCInspector.java:284 - G1 Young Generation GC in 286ms.  G1 Eden Space: 1577058304 -> 0; G1 Old Gen: 7451103472 -> 4596956920; G1 Survivor Space: 134217728 -> 150994944;
INFO  [Service Thread] 2018-05-12 18:03:57,652 GCInspector.java:284 - G1 Young Generation GC in 213ms.  G1 Eden Space: 1560281088 -> 0; G1 Old Gen: 4596956920 -> 4135583472; G1 Survivor Space: 150994944 -> 134217728;
INFO  [CompactionExecutor:172351] 2018-05-12 18:21:29,098 AutoSavingCache.java:399 - Saved CounterCache (258640 items) in 298 ms
INFO  [CompactionExecutor:172352] 2018-05-12 18:21:29,979 AutoSavingCache.java:399 - Saved KeyCache (788252 items) in 1282 ms

在其他几天我有不同节点拥有100%cpu,有什么想法,它不是同一节点吗?

Heap size Image

Table: abc
                SSTable count: 14
                Space used (live): 3 960 499 258
                Space used (total): 3960499258
                Space used by snapshots (total): 0
                Off heap memory used (total): 8427312
                SSTable Compression Ratio: 0.25806487155056185
                Number of keys (estimate): 379760
                Memtable cell count: 172086
                Memtable data size: 1253025
                Memtable off heap memory used: 4654441
                Memtable switch count: 192
                Local read count: 33923975
                Local read latency: 15.081 ms
                Local write count: 30595833
                Local write latency: 0.144 ms
                Pending flushes: 0
                Bloom filter false positives: 18478
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 1523216
                Bloom filter off heap memory used: 1523104
                Index summary off heap memory used: 459327
                Compression metadata off heap memory used: 1790440
                Compacted partition minimum bytes: 30
                Compacted partition maximum bytes: 4055269
                Compacted partition mean bytes: 11408
                Average live cells per slice (last five minutes): 1.0
                Maximum live cells per slice (last five minutes): 1
                Average tombstones per slice (last five minutes): 178.40981886703793
                Maximum tombstones per slice (last five minutes): 258



Table: xyz
                SSTable count: 8
                Space used (live): 2034316302
                Space used (total): 2034316302
                Space used by snapshots (total): 0
                Off heap memory used (total): 4289964
                SSTable Compression Ratio: 0.24435625500089553
                Number of keys (estimate): 260117
                Memtable cell count: 133822
                Memtable data size: 832669
                Memtable off heap memory used: 2215708
                Memtable switch count: 175
                Local read count: 10034316
                Local read latency: 11.653 ms
                Local write count: 27984185
                Local write latency: 0.075 ms
                Pending flushes: 0
                Bloom filter false positives: 261009
                Bloom filter false ratio: 0.04762
                Bloom filter space used: 1007024
                Bloom filter off heap memory used: 1006960
                Index summary off heap memory used: 200632
                Compression metadata off heap memory used: 866664
                Compacted partition minimum bytes: 61
                Compacted partition maximum bytes: 5839588
                Compacted partition mean bytes: 8883
                Average live cells per slice (last five minutes): 1.0
                Maximum live cells per slice (last five minutes): 1
                Average tombstones per slice (last five minutes): 28.292941014323677
                Maximum tombstones per slice (last five minutes): 60



Table: zzz
                SSTable count: 8
                Space used (live): 1542111866
                Space used (total): 1542111866
                Space used by snapshots (total): 0
                Off heap memory used (total): 14022908
                SSTable Compression Ratio: 0.5677972573793397
                Number of keys (estimate): 8670916
                Memtable cell count: 0
                Memtable data size: 0
                Memtable off heap memory used: 0
                Memtable switch count: 0
                Local read count: 0
                Local read latency: NaN ms
                Local write count: 0
                Local write latency: NaN ms
                Pending flushes: 0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 10623352
                Bloom filter off heap memory used: 10623288
                Index summary off heap memory used: 3157452
                Compression metadata off heap memory used: 242168
                Compacted partition minimum bytes: 36
                Compacted partition maximum bytes: 268650950
                Compacted partition mean bytes: 242
                Average live cells per slice (last five minutes): NaN
                Maximum live cells per slice (last five minutes): 0
                Average tombstones per slice (last five minutes): NaN
                Maximum tombstones per slice (last five minutes): 0  

现在负载已经下降。只是在加载abc表统计数据后进行比较:

Table: abc
                SSTable count: 14
                Space used (live): 3960499258
                Space used (total): 3960499258
                Space used by snapshots (total): 0
                Off heap memory used (total): 24792685
                SSTable Compression Ratio: 0.25806487155056185
                Number of keys (estimate): 380267
                Memtable cell count: 780522
                Memtable data size: 3746628
                Memtable off heap memory used: 21019814
                Memtable switch count: 192
                Local read count: 33927339
                Local read latency: 9.319 ms
                Local write count: 30598848
                Local write latency: 0.093 ms
                Pending flushes: 0
                Bloom filter false positives: 18478
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 1523216
                Bloom filter off heap memory used: 1523104
                Index summary off heap memory used: 459327
                Compression metadata off heap memory used: 1790440
                Compacted partition minimum bytes: 30
                Compacted partition maximum bytes: 4055269
                Compacted partition mean bytes: 11408
                Average live cells per slice (last five minutes): 1.0
                Maximum live cells per slice (last five minutes): 1
                Average tombstones per slice (last five minutes): 179.0
                Maximum tombstones per slice (last five minutes): 179

0 个答案:

没有答案