cassandra非常高的CPU率

时间:2016-04-19 12:08:00

标签: cassandra cassandra-3.0

我正在使用Cassandra 3.0.5版,最近刚刚从3.0.4升级。

在非常低的CPU利用率100%的查询速度下。

看看wather,cpu几乎都被用户进程占用了:

enter image description here

打击是顶级-H信息(24 cpu核心机):

top - 19:54:50 up 21 days,  3:06,  7 users,  load average: 25.15, 26.44, 26.66
Tasks: 987 total,  29 running, 958 sleeping,   0 stopped,   0 zombie
Cpu(s): 98.7%us,  0.1%sy,  0.0%ni,  1.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  66068260k total, 61686668k used,  4381592k free,   423456k buffers
Swap: 50331640k total,        0k used, 50331640k free, 41947168k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                         
51174 cassandr  20   0 42.7g  16g 3.2g R 96.7 26.1  90:42.09 java                                                                                                                                            
51166 cassandr  20   0 42.7g  16g 3.2g R 95.1 26.1  82:13.11 java                                                                                                                                            
51179 cassandr  20   0 42.7g  16g 3.2g R 93.8 26.1  93:40.75 java                                                                                                                                            
51164 cassandr  20   0 42.7g  16g 3.2g R 92.8 26.1  75:29.43 java                                                                                                                                            
51171 cassandr  20   0 42.7g  16g 3.2g R 92.5 26.1  91:06.15 java                                                                                                                                            
51180 cassandr  20   0 42.7g  16g 3.2g R 90.2 26.1  94:27.94 java                                                                                                                                            
51173 cassandr  20   0 42.7g  16g 3.2g R 89.9 26.1  86:18.13 java                                                                                                                                            
51176 cassandr  20   0 42.7g  16g 3.2g R 89.5 26.1  93:53.24 java                                                                                                                                            
51168 cassandr  20   0 42.7g  16g 3.2g R 87.3 26.1  83:39.00 java                                                                                                                                            
51446 cassandr  20   0 42.7g  16g 3.2g R 86.6 26.1  96:57.18 java                                                                                                                                            
51455 cassandr  20   0 42.7g  16g 3.2g R 85.6 26.1  68:25.76 java                                                                                                                                            
51183 cassandr  20   0 42.7g  16g 3.2g R 82.7 26.1  93:11.25 java                                                                                                                                            
51181 cassandr  20   0 42.7g  16g 3.2g R 82.0 26.1  93:10.51 java                                                                                                                                            
51448 cassandr  20   0 42.7g  16g 3.2g R 81.0 26.1  94:01.51 java                                                                                                                                            
51444 cassandr  20   0 42.7g  16g 3.2g R 79.7 26.1  95:20.24 java                                                                                                                                            
51182 cassandr  20   0 42.7g  16g 3.2g R 79.4 26.1  92:09.21 java                                                                                                                                            
51449 cassandr  20   0 42.7g  16g 3.2g R 78.8 26.1  93:27.02 java                                                                                                                                            
51447 cassandr  20   0 42.7g  16g 3.2g R 78.4 26.1  91:46.38 java                                                                                                                                            
51453 cassandr  20   0 42.7g  16g 3.2g R 77.1 26.1  87:55.30 java                                                                                                                                            
51170 cassandr  20   0 42.7g  16g 3.2g R 75.2 26.1  86:45.33 java                                                                                                                                            
51445 cassandr  20   0 42.7g  16g 3.2g R 73.5 26.1  95:42.81 java                                                                                                                                            
51172 cassandr  20   0 42.7g  16g 3.2g R 73.2 26.1  90:37.34 java                                                                                                                                            
51452 cassandr  20   0 42.7g  16g 3.2g R 72.9 26.1  94:18.97 java                                                                                                                                            
51451 cassandr  20   0 42.7g  16g 3.2g R 72.2 26.1  92:12.47 java                                                                                                                                            
51178 cassandr  20   0 42.7g  16g 3.2g R 71.2 26.1  93:16.02 java                                                                                                                                            
51165 cassandr  20   0 42.7g  16g 3.2g R 69.6 26.1  79:04.09 java                                                                                                                                            
51169 cassandr  20   0 42.7g  16g 3.2g R 66.7 26.1  78:41.26 java                                                                                                                                            
51175 cassandr  20   0 42.7g  16g 3.2g R 63.1 26.1  94:24.96 java                                                                                                                                            
51443 cassandr  20   0 42.7g  16g 3.2g S 53.9 26.1  93:11.98 java 

然后从jstack看到线程信息:

"SharedPool-Worker-26" #301 daemon prio=5 os_prio=0 tid=0x00007ff8b83362e0 nid=0xc8f8 runnable [0x00007ff8867ef000]
   java.lang.Thread.State: RUNNABLE
    at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:110)
    at org.apache.cassandra.index.internal.composites.CompositesSearcher$1.prepareNext(CompositesSearcher.java:130)
    at org.apache.cassandra.index.internal.composites.CompositesSearcher$1.hasNext(CompositesSearcher.java:83)
    at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
    at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295)
    at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134)
    at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127)
    at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
    at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65)
    at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289)
    at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
    at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
    at java.lang.Thread.run(Thread.java:745)

或者这个:

org.apache.cassandra.utils.MurmurHash.hash3_x64_128(MurmurHash.java:191)
org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:181)
org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:53)
org.apache.cassandra.db.PartitionPosition$ForKey.get(PartitionPosition.java:49)
org.apache.cassandra.db.marshal.PartitionerDefinedOrder.compareCustom(PartitionerDefinedOrder.java:93)
org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:158)
org.apache.cassandra.db.ClusteringComparator.compareComponent(ClusteringComparator.java:166)
org.apache.cassandra.db.ClusteringComparator.compare(ClusteringComparator.java:137)
org.apache.cassandra.db.ClusteringComparator.compare(ClusteringComparator.java:126)
org.apache.cassandra.db.ClusteringComparator.compare(ClusteringComparator.java:44)
org.apache.cassandra.utils.MergeIterator$Candidate.compareTo(MergeIterator.java:378)
org.apache.cassandra.utils.MergeIterator$ManyToOne.replaceAndSink(MergeIterator.java:266)
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:428)
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:288)
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108)
org.apache.cassandra.index.internal.composites.CompositesSearcher$1.prepareNext(CompositesSearcher.java:130)
org.apache.cassandra.index.internal.composites.CompositesSearcher$1.hasNext(CompositesSearcher.java:83)
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295)
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134)
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127)
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65)
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289)
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
java.lang.Thread.run(Thread.java:745)

我发现繁忙的运行线程全部来自“SharedPool-Worker”。

它们都在org.apache.cassandra.db.transform.BaseRows.hasNext或比方法更深。

有人遇到过这个问题吗?这是一个错误吗?

1 个答案:

答案 0 :(得分:0)

不是cassandra问题,使用二级索引是我的错。 我在我的应用程序中发现了这个cql:SELECT * FROM userlabel where phone =''。 由于表“userlabel”中有许多空手机,这个cql会选出太多行,并且它们被编入索引。 它也可能是问题“查询第二索引时堆栈溢出”CASSANDRA-11304的主要原因,因为需要查询太多行,然后通过递归调用发生错误。