Elasticsearch-高CPU使用率,可在负载测试时进行简单查询

时间:2019-03-19 12:39:07

标签: elasticsearch

我有一个Elasticsearch索引,其中3个分片上包含约9300万份文档(68.8GB)。使用一个特定查询进行负载测试(300个并发请求)时,CPU使用率猛增至100%。查询非常简单:

{"query": 
    {"bool": 
        {"must": [
            {"match": {"release_title": "Spider - Singles Collection 1976-86"}}, 
            {"match": {"track_title": "TALKIN BOUT ROCK N ROLL (New Version)"}}
                 ]
        }
    }
}

但是,如果我使用不同的字符串进行相同类型的查询,则CPU使用率是正常的。例如,以下查询的CPU使用率是正常的:

{"query": 
    {"bool": 
        {"must": [
            {"match": {"release_title": "Spark Seeker"}}, 
            {"match": {"track_title": "Live Like A Warrior"}}
                 ]
        }
    }
}

这是Hot Threads的输出(我是ES的初学者,不确定如何解释):

Hot threads at 2019-03-19T12:21:30.840Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=t
rue:

   81.4% (407.1ms out of 500ms) cpu usage by thread 'elasticsearch[3jOzxpW][search][T#2]'
     5/10 snapshots sharing following 21 elements
       org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation
.java:66)
       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     5/10 snapshots sharing following 17 elements
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)

   80.9% (404.2ms out of 500ms) cpu usage by thread 'elasticsearch[3jOzxpW][search][T#3]'
     7/10 snapshots sharing following 21 elements
       org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:66)
       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:200)
       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 23 elements
       org.apache.lucene.search.DisiPriorityQueue.downHeap(DisiPriorityQueue.java:158)
       org.apache.lucene.search.DisiPriorityQueue.updateTop(DisiPriorityQueue.java:119)
       org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:67)
       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     unique snapshot
       org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:66)
       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)

   78.6% (392.8ms out of 500ms) cpu usage by thread 'elasticsearch[3jOzxpW][search][T#13]'
     4/10 snapshots sharing following 20 elements
       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:200)
       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 21 elements
       org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:66)
       org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
       org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 24 elements
       org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:198)
       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
       org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)
       org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:226)
       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
       org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)
       org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:226)
       org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
       org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
       org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:108)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 17 elements
       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
       org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
       org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)

系统具有14个核心和40GB的RAM。我已将JVM堆大小设置为16GB。有人可以帮我找出问题所在吗?

0 个答案:

没有答案