我们在生产环境中部署了WSO2 API Manager 2.2.0。
我们的架构: 我们在DMZ中部署了一个API网关实例。此API网关实例通过负载平衡器将所有有效的API请求转发到内部“多合一” WSO2 API管理器实例(如果Prod实例不可用,则将请求路由到DR实例)。多合一实例将Oracle数据库用于API,注册表,用户管理,Message Broker和统计。
我们每小时都会发生一次日志错误。该错误的影响似乎是,API管理器无法在大约1分钟的时间内与端点进行通信,直到问题似乎自行解决为止。因此,WSO2将以“发件人错误”消息回应科研人员,并且该API将被暂停30秒钟。
任何人都可以建议造成此问题的原因是什么
每小时发生一次错误:
TID: [-1] [] [2019-01-30 01:00:12,156] WARN {org.wso2.andes.client.protocol.AMQProtocolHandler} - Timed out while waiting for heartbeat from peer. {org.wso2.andes.client.protocol.AMQProtocolHandler}
TID: [-1] [] [2019-01-30 01:00:12,272] ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina {org.wso2.andes.transport.network.mina.MinaNetworkHandler}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:748)
TID: [-1] [] [2019-01-30 01:00:12,280] ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina {org.wso2.andes.transport.network.mina.MinaNetworkHandler}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:748)
TID: [-1] [] [2019-01-30 01:00:12,519] ERROR {org.wso2.andes.server.protocol.AMQProtocolEngine} - IOException caught in/<ip-address>:49069(user), session closed implictly: java.io.IOException: Connection reset by peer {org.wso2.andes.server.protocol.AMQProtocolEngine}
TID: [-1] [] [2019-01-30 01:00:12,520] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:49069(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,520] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '246' on channel [/<ip-address>:49069(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,456] ERROR {org.wso2.andes.server.protocol.AMQProtocolEngine} - IOException caught in/<ip-address>:7076(user), session closed implictly: java.io.IOException: Connection reset by peer {org.wso2.andes.server.protocol.AMQProtocolEngine}
TID: [-1] [] [2019-01-30 01:00:12,522] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:7076(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,526] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '44' on channel [/<ip-address>:7076(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,530] ERROR {org.wso2.andes.client.state.AMQStateManager} - No Waiters for error saving as last error:Exception thrown against AMQConnection:
Host: <ip-address>
Port: 5672
Virtual Host: carbon
Client ID: clientid
Active session count: 1: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted. {org.wso2.andes.client.state.AMQStateManager}
TID: [-1] [] [2019-01-30 01:00:12,533] ERROR {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - JMS Connection failed : Exception thrown against AMQConnection:
Host: <ip-address>
Port: 5672
Virtual Host: carbon
Client ID: clientid
Active session count: 1: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted. - shutting down worker tasks {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:12,534] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Reconnection attempt : 1 for Siddhi-JMS-Consumer {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:12,536] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:29986(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,549] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '1' on channel [/<ip-address>:29986(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,552] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=386@NODE<server>/<ip-address>/T=1548808789236/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,557] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=385@<server>/<ip-address>/T=1548808788762/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,557] INFO {org.wso2.andes.kernel.OrphanedMessageHandler} - Purging messages of this node persisted under throttleData {org.wso2.andes.kernel.OrphanedMessageHandler}
TID: [-1] [] [2019-01-30 01:00:12,773] INFO {org.wso2.andes.kernel.MessagingEngine} - Purged messages of destination throttleData {org.wso2.andes.kernel.MessagingEngine}
TID: [-1] [] [2019-01-30 01:00:12,780] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=384@<server>/<ip-address>/T=1548806362449/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,782] INFO {org.wso2.andes.kernel.OrphanedMessageHandler} - Purging messages of this node persisted under throttleData {org.wso2.andes.kernel.OrphanedMessageHandler}
TID: [-1] [] [2019-01-30 01:00:12,791] INFO {org.wso2.andes.kernel.MessagingEngine} - Purged messages of destination throttleData {org.wso2.andes.kernel.MessagingEngine}
TID: [-1] [] [2019-01-30 01:00:13,517] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:49069) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:13,751] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:29986) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:13,926] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:7076) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:17,607] WARN {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Unable to shutdown all polling tasks of Siddhi-JMS-Consumer {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:17,609] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Task manager for jms consumer 1000 shutdown {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:17,612] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Task manager for Siddhi-JMS-Consumer [re-]initialized {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1234] [] [2019-01-30 01:00:17,760] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - SASL Mechanism selected: PLAIN {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,760] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Locale selected: en_US {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,782] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Connected as: user{org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,783] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Framesize set to 65535 {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,873] INFO {org.wso2.andes.server.handler.ChannelOpenHandler} - Connecting to: carbon {org.wso2.andes.server.handler.ChannelOpenHandler}
TID: [-1234] [] [2019-01-30 01:00:17,875] INFO {org.wso2.andes.kernel.AndesChannel} - Channel created (ID: <ip-address>:31136) {org.wso2.andes.kernel.AndesChannel}
TID: [-1234] [] [2019-01-30 01:00:17,957] INFO {org.wso2.andes.server.handler.QueueDeclareHandler} - Queue tmp_ip-address_31136_1 bound to default exchange(<<default>>) {org.wso2.andes.server.handler.QueueDeclareHandler}
TID: [-1234] [] [2019-01-30 01:00:17,958] INFO {org.wso2.andes.server.handler.QueueDeclareHandler} - Queue tmp_ip-address_31136_1 declared successfully {org.wso2.andes.server.handler.QueueDeclareHandler}
TID: [-1234] [] [2019-01-30 01:00:18,074] INFO {org.wso2.andes.server.handler.QueueBindHandler} - Binding queue tmp_ip-address_31136_1 to exchange TopicExchange[amq.topic] with routing key throttleData {org.wso2.andes.server.handler.QueueBindHandler}
TID: [-1] [] [2019-01-30 01:00:18,615] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Reconnection attempt: 1 for Siddhi-JMS-Consumer was successful! {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:20,428] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local subscription ADDED [throttleData]ID=387@<server>/<ip-address>/T=1548810020424/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=true {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:05:28,619] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}