我们正在运行Infinispan 7.2.5,在REPL集群中有3个实例。 Spark Client使用HotRod连接到群集。
突然,视图被更新,一个实例也从集群和客户端视图中删除。 实例是up&正在运行,但无法连接到其他实例,并且正在计时超时。
什么可能导致实例离开群集而不让它加入?如果有人可以就此提供一些见解。
同时,CPU在实例上太高了。是因为重新加入尝试(如果实际需要大量CPU)或其他原因可能是什么原因?
使用HotRod连接到Infinispan的客户端在以下情况下进行了删除:
WARN (ClientListenerNotifier.java:266) - ISPN004039: Unable to complete reading event from server null
java.nio.channels.IllegalBlockingModeException
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARN (ClientListenerNotifier.java:266) - ISPN004039: Unable to complete reading event from server null
java.nio.channels.IllegalBlockingModeException
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
在同一时间,Infinispan服务器实例收到以下日志:
WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p3-t13) ISPN000071: Caught exception when handling command StateResponseCommand{cache=AsrlEnbTopologyCache, origin=asr-1-asrltopologyservice-24247, topologyId=9}: org.infinispan.util.concurrent.TimeoutException: Timed out applying state
at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:542) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:62) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:32) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
然而,似乎与群集分离的其他可用实例已收到以下日志:
ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p2-t12) Failed to send entries to node asr-2-asrltopologyservice-2286 : Node asr-2-asrltopologyservice-2286 timed out: org.infinispan.util.concurrent.TimeoutException: Node asr-2-asrltopologyservice-2286 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:239) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntry(OutboundTransferTask.java:195) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:149) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
Caused by: org.jgroups.TimeoutException: timeout waiting for response from asr-2-asrltopologyservice-2286, request: org.jgroups.blocks.UnicastRequest@6ac2f8a3, req_id=1067, mode=GET_ALL, target=asr-2-asrltopologyservice-2286
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427) [jgroups-3.6.2.Final.jar:3.6.2.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
... 12 more
ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p2-t14) Failed to send entries to node asr-2-asrltopologyservice-2286 : Node asr-2-asrltopologyservice-2286 timed out: org.infinispan.util.concurrent.TimeoutException: Node asr-2-asrltopologyservice-2286 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:239) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntry(OutboundTransferTask.java:195) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:149) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
Caused by: org.jgroups.TimeoutException: timeout waiting for response from asr-2-asrltopologyservice-2286, request: org.jgroups.blocks.UnicastRequest@1984efac, req_id=1069, mode=GET_ALL, target=asr-2-asrltopologyservice-2286
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427) [jgroups-3.6.2.Final.jar:3.6.2.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
... 12 more