通过WSO2 API Manager日志，我们注意到很少出现调用返回为HTTP 500错误并且在客户端后续重试时成功。这不会一直发生（约为5％）。所有这些事件都来自同一个源IP。进一步看，这些使用HTTP 500失败的调用永远不会被发送到后端端点进行实际处理 - 因此，这些调用都被WSO2 API Manager本身失败了。
我们已经验证了CPU，WSO2的内存处于正常工作水平。我们的WSO2 APIM环境没有启用授权令牌（简单通过节流阀） - 因此可以计算出来。
从日志中，几乎立即返回错误500，并且通常在200ms内重试。所有这些错误（大约49个）都在同一个小时内发生。
我们也无法验证确切的错误消息，因为API调用来自我们的客户。
感谢您是否可以列出所有可能性，因为在这种情况下WSO2可能会返回HTTP 500。否则，我们如何进一步调查方案。感谢。
以下是相关日志
WSO2 APIM AWS ELB访问日志
$ grep 'fromIndex=394501&toIndex=395000' *.log
WSO2APIM_ELBACESS.log:2016-12-06T08:37:37.905822Z wso2-oss-server x.x.x.x:63598 x.x.x.98:8280 0.000042 0.001862 0.000023 500 500 0 293 "GET https://wso2apimanager:443/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "RestSharp/x.x.x.x" ECDHE-RSA-AES128-SHA TLSv1
WSO2APIM_ELBACESS.log:2016-12-06T08:37:38.102539Z wso2-oss-server x.x.x.x:63598 x.x.x.120:8280 0.000041 0.035397 0.000025 200 200 0 33 "GET https://wso2apimanager:443/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "RestSharp/x.x.x.x" ECDHE-RSA-AES128-SHA TLSv1
WSO2 APIM http访问日志
$ grep 'fromIndex=394501&toIndex=395000' *.log
http_access_2016-12-06.log_m1:x.x.x.x x.x.x.140 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "RestSharp/x.x.x.x"
http_access_2016-12-06.log_m1:x.x.x.x x.x.x.72 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "Synapse-PT-HttpComponents-NIO"
http_access_2016-12-06.log_m2:x.x.x.x x.x.x.140 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "RestSharp/x.x.x.x"
后端AWS ELB访问日志
$ grep 'fromIndex=394501&toIndex=395000' *.log
BACKEND_ELBACCESS.log:2016-12-06T08:37:38.104641Z vpc-connect-api x.x.x.120:46946 x.x.x.23:80 0.000045 0.032717 0.000022 200 200 0 33 "GET http://backendserver:80/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "Synapse-PT-HttpComponents-NIO" - -
WSO2 API Manager 1（wso2carbon.log）
TID: [-1234] [] [2016-12-06 08:34:47,427] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected {org.wso2.carbon.databridge.core.DataBridge}
TID: [-1234] [] [2016-12-06 08:35:18,069] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:35:47,850] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1] [] [2016-12-06 08:37:13,485] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16702: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:13,486] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,486] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_28 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:43 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,486] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,487] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566702 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:37:13,961] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:18,342] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:18,007] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:40:17,933] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:41:17,725] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:43:17,811] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:45:17,892] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:47:18,363] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:50:47,850] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:51:17,864] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:53:17,753] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:55:18,025] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:57:18,333] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:59:17,949] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
WSO2 API Manager 2（wso2carbon.log）
TID: [-1234] [] [2016-12-06 08:34:54,620] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected {org.wso2.carbon.databridge.core.DataBridge}
TID: [-1234] [] [2016-12-06 08:36:18,043] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,674] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16760: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:13,692] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,693] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_28 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:43 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,693] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,694] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566592 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:37:14,331] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1] [] [2016-12-06 08:37:14,620] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16759: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:14,621] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:14,621] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:44 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:14,621] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:14,818] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:15,255] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:29,543] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,137] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,568] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,990] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:36,409] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:37,899] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:38,327] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:41,129] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:41,545] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:42,969] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:38:09,156] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:38:17,817] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:39:16,163] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566604 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1] [] [2016-12-06 08:39:19,169] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566640 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:39:19,648] ERROR {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Failed to submit the response {org.apache.synapse.transport.passthru.PassThroughHttpSender}
java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:19,649] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/json,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Failed to submit the response
at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:613)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
Caused by: java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:19,650] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 0, ERROR_MESSAGE = Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/json,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:19,688] WARN {org.apache.synapse.core.axis2.Axis2Sender} - Trying to send a response to an already responded client request - Rest API Context : /rest {org.apache.synapse.core.axis2.Axis2Sender}
TID: [-1] [] [2016-12-06 08:39:20,249] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : internal-vpc-connect-api-1883227725.ap-southeast-1.elb.amazonaws.com/x.x.x.43:80 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2016-12-06 08:39:20,252] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:20,254] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:39:50 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:20,254] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:20,254] ERROR {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Failed to submit the response {org.apache.synapse.transport.passthru.PassThroughHttpSender}
java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:20,255] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Accept-Encoding:gzip, deflate,Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/xml,password:Hjfawx99@@,username:lynn@heatherjames.com,X-Forwarded-For:76.79.103.18,X-Forwarded-Port:443,X-Forwarded-Proto:https,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><am:fault xmlns:am="http://wso2.org/apimanager"><am:code>101503</am:code><am:type>Status report</am:type><am:message>Runtime Error</am:message><am:description>Error connecting to the back end</am:description></am:fault></soapenv:Body></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Failed to submit the response
at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:613)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
Caused by: java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:20,255] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:20,256] WARN {org.apache.synapse.core.axis2.Axis2Sender} - Trying to send a response to an already responded client request - Rest API Context : /rest {org.apache.synapse.core.axis2.Axis2Sender}
TID: [-1] [] [2016-12-06 08:39:22,936] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : internal-vpc-connect-api-1883227725.ap-southeast-1.elb.amazonaws.com/x.x.x.43:80 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2016-12-06 08:39:22,942] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:22,945] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:39:52 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:22,945] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:23,363] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
x 34 of similiar logs
TID: [-1234] [] [2016-12-06 08:39:52,303] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:53,910] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:42:19,456] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:42:56,120] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:44:18,039] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:46:17,939] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:46:36,177] WARN {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Invalid session id for thrift authenticator. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2016-12-06 08:46:36,177] ERROR {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Error in invoking validate key via thrift.. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2016-12-06 08:46:36,177] WARN {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient} - Login failed.. Authenticating again.. {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient}
TID: [-1234] [] [2016-12-06 08:46:36,231] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin@carbon.super [-1234]' logged in at [2016-12-06 08:46:36,231+0000] from IP address {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1234] [] [2016-12-06 08:48:17,776] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:49:18,033] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:50:17,748] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:52:18,419] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:54:18,051] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:56:17,815] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:57:56,120] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:58:18,054] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}