Question

我们的3节点集群中的一个节点已关闭并且在检查日志文件时，它显示以下消息

INFO  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:32,891  AbstractMetrics.java:114 - Cannot record QUEUE latency of 11 minutes because higher than 10 minutes.
INFO  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:33,233  AbstractMetrics.java:114 - Cannot record QUEUE latency of 10 minutes because higher than 10 minutes.
WARN  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:33,398  Worker.java:99 - Interrupt/timeout detected.
java.util.concurrent.BrokenBarrierException: null
at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:200) ~[na:1.7.0_79]
at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355) ~[na:1.7.0_79]
at com.datastax.bdp.concurrent.FlushTask.bulkSync(FlushTask.java:76) ~[dse-core-4.8.3.jar:4.8.3]
at com.datastax.bdp.concurrent.Worker.run(Worker.java:94) ~[dse-core-4.8.3.jar:4.8.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
WARN  [keyspace.core Index WorkPool work thread-2] 2016-09-14 14:05:33,398  Worker.java:99 - Interrupt/timeout detected.
java.util.concurrent.BrokenBarrierException: null
at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:200) ~[na:1.7.0_79]
at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355) ~[na:1.7.0_79]
at com.datastax.bdp.concurrent.FlushTask.bulkSync(FlushTask.java:76) ~[dse-core-4.8.3.jar:4.8.3]
at com.datastax.bdp.concurrent.Worker.run(Worker.java:94) ~[dse-core-4.8.3.jar:4.8.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
INFO  [keyspace.core Index WorkPool work thread-4] 2016-09-14 14:05:33,720  AbstractMetrics.java:114 - Cannot record QUEUE latency of 13 minutes because higher than 10 minutes.
INFO  [keyspace.core Index WorkPool work thread-4] 2016-09-14 14:05:33,721  AbstractMetrics.java:114 - Cannot record QUEUE latency of 13 minutes because higher than 10 minutes.

节点配置为8个CPU，32 GB RAM，500 GB磁盘空间。只有一个特定节点出现故障的原因是什么？

Answer 1

所以我将在这里回答一些一般信息，你的情况可能会更复杂。对于Solr节点，32GB RAM可能不够大;使用Java 1.8上的G1收集器对于堆大小超过26GB的Solr来说已经证明更好。

我也不确定你的堆大小，JVM设置和你有多少个solr核心。但是，当节点忙于索引并且它试图跟上时，我已经看到了类似的错误。根据我的经验，在Solr节点上看到的最常见问题之一是max_solr_concurrency_per_core中默认（注释掉）dse.yaml。这通常会将索引线程的数量分配给CPU核心的数量，并且为了进一步加剧问题，您可能会看到8个核心，但如果您有HT，那么它实际上可能有4个物理核心。

检查您的dse.yaml并确保将其设置为num physcal cpu cores / num of solr cores至少为2。这可能指数较慢，但您应该消除节点的压力。

我建议这个有用的博客作为调整DSE Solr的良好开端：

http://www.datastax.com/dev/blog/tuning-dse-search

还有关于这个主题的文档：

https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/srch/srchTune.html

无法记录n分钟的QUEUE延迟 - DSE

1 个答案: