我有一个Elasticsearch索引,其中3个分片上包含约9300万份文档(68.8GB)。使用一个特定查询进行负载测试(300个并发请求)时,CPU使用率猛增至100%。查询非常简单:
{"query":
{"bool":
{"must": [
{"match": {"release_title": "Spider - Singles Collection 1976-86"}},
{"match": {"track_title": "TALKIN BOUT ROCK N ROLL (New Version)"}}
]
}
}
}
但是,如果我使用不同的字符串进行相同类型的查询,则CPU使用率是正常的。例如,以下查询的CPU使用率是正常的:
{"query":
{"bool":
{"must": [
{"match": {"release_title": "Spark Seeker"}},
{"match": {"track_title": "Live Like A Warrior"}}
]
}
}
}
这是Hot Threads的输出(我是ES的初学者,不确定如何解释):
Hot threads at 2019-03-19T12:21:30.840Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=t
rue:
81.4% (407.1ms out of 500ms) cpu usage by thread 'elasticsearch[3jOzxpW][search][T#2]'
5/10 snapshots sharing following 21 elements
org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation
.java:66)
org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
5/10 snapshots sharing following 17 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
80.9% (404.2ms out of 500ms) cpu usage by thread 'elasticsearch[3jOzxpW][search][T#3]'
7/10 snapshots sharing following 21 elements
org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:66)
org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:200)
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 23 elements
org.apache.lucene.search.DisiPriorityQueue.downHeap(DisiPriorityQueue.java:158)
org.apache.lucene.search.DisiPriorityQueue.updateTop(DisiPriorityQueue.java:119)
org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:67)
org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
unique snapshot
org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:66)
org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
78.6% (392.8ms out of 500ms) cpu usage by thread 'elasticsearch[3jOzxpW][search][T#13]'
4/10 snapshots sharing following 20 elements
org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:200)
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 21 elements
org.apache.lucene.search.DisjunctionDISIApproximation.advance(DisjunctionDISIApproximation.java:66)
org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:202)
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:219)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 24 elements
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:198)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:226)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:57)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:226)
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:108)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 17 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:379)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:259)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:273)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:300)
org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
系统具有14个核心和40GB的RAM。我已将JVM堆大小设置为16GB。有人可以帮我找出问题所在吗?