我们正在使用TSDB(net.opentsdb.core.TSDB)Java API从OpenTSDB保存和检索数据点。当区域开始拆分时,java API无法保存数据点,甚至线程在保存数据点时也会被阻塞。即使区域恢复联机后,它仍然无法保存数据点,并且再也无法恢复。
Thread dump of blocked threads on save API :-
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.stumbleupon.async.Deferred.doJoin(Deferred.java:1138)
- locked <0x00000006acc5d570> (a com.stumbleupon.async.Deferred$Signal)
at com.stumbleupon.async.Deferred.joinUninterruptibly(Deferred.java:1064)
at net.opentsdb.uid.UniqueId.getOrCreateId(UniqueId.java:673)
at net.opentsdb.core.IncomingDataPoints.rowKeyTemplate(IncomingDataPoints.java:139)
at net.opentsdb.core.TSDB.addPointInternal(TSDB.java:963)
at net.opentsdb.core.TSDB.addPoint(TSDB.java:873)
at com.appdynamics.pi.tsd.client.OpenTSDBClient.addPoint(OpenTSDBClient.java:76)
at com.appdynamics.pi.tsd.dao.OpentsdbAdapter.saveDataPoints(OpentsdbAdapter.java:61)
at com.appdynamics.pi.tsd.dao.TimeSeriesDaoOpenTSDB.saveContextualAnomaly(TimeSeriesDaoOpenTSDB.java:50)
at com.appdynamics.pi.tsd.processor.ContextualMessageProcessor.saveContextualAnomalyEvent(ContextualMessageProcessor.java:69)
at com.appdynamics.pi.tsd.processor.ContextualMessageProcessor.process(ContextualMessageProcessor.java:41)
at com.appdynamics.pi.tsd.processor.ContextualMessageProcessor.process(ContextualMessageProcessor.java:24)
at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
at org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403)
at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317)
at org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942)
在应用程序日志中,我们看到以下消息:-
由于NSRE,现在有1500个RPC待处理
由于NSRE,现在有2000个RPC待处理