由于Cassandra连接问题，API Key上的代理API超时

时间：2014-04-02 12:11:43

标签： apigee

我很感激帮助分手并解决这个问题：

我正在 频繁 期间，所有代理API都挂起，跟踪显示“???”对于请求的HTTP状态代码，我在30秒后收到此响应：

 Status Code: 504 Gateway Timeout 
 Content-Length: 177 
 Content-Type: text/xml; charset=UTF-8 

<?xml version='1.0' encoding='UTF-8'?><fault><faultstring>The Service is temporarily unavailable</faultstring><detail><errorcode>SERVICE_UNAVAILABLE</errorcode></detail></fault>

这是我在system.log中看到的所有三个Cassandra服务器

>     2014-04-01 14:29:20,124 org: env: Apigee-Main-36 ERROR m.p.c.c.c.HThriftClient - HThriftClient.close() : Could not flush
> transport (to be expected if the pool is shutting down) in close for
> client: CassandraClient<10.49.192.52:9160-829>
>     org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
>       at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
> ~[libthrift-0.7.0.jar:0.7.0]
>       at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
> ~[libthrift-0.7.0.jar:0.7.0]
>       at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:125)
> [hector-core-1.1-3.jar:na]
>       at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:38)
> [hector-core-1.1-3.jar:na]
>       at me.prettyprint.cassandra.connection.HConnectionManager.closeClient(HConnectionManager.java:325)
> [hector-core-1.1-3.jar:na]
>       at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:273)
> [hector-core-1.1-3.jar:na]
>       at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113)
> [hector-core-1.1-3.jar:na]
>       at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.sliceInternal(ThriftColumnFamilyTemplate.java:88)
> [hector-core-1.1-3.jar:na]
>       at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.doExecuteSlice(ThriftColumnFamilyTemplate.java:46)
> [hector-core-1.1-3.jar:na]
>       at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.queryColumns(ColumnFamilyTemplate.java:113)
> [hector-core-1.1-3.jar:na]
>       at com.apigee.datastore.client.CassandraClient.get(CassandraClient.java:169)
> [datastore-1.0.0.jar:na]
>       at com.apigee.keymanagement.dao.nosql.impl.AppDaoImpl.getCredential(AppDaoImpl.java:123)
> [keymanagement-1.0.0.jar:na]
>       at com.apigee.keymanagement.dao.nosql.impl.AppDaoImpl.getConsumerKeyStatus(AppDaoImpl.java:77)
> [keymanagement-1.0.0.jar:na]
>       at com.apigee.keymanagement.util.ResourceUtil.validateConsumerKey(ResourceUtil.java:490)
> [keymanagement-1.0.0.jar:na]
>       at com.apigee.keymanagement.util.ResourceUtil.validateConsumerKey(ResourceUtil.java:475)
> [keymanagement-1.0.0.jar:na]
>       at com.apigee.keymanagement.util.ResourceUtil.getConsumerDetails(ResourceUtil.java:526)
> [keymanagement-1.0.0.jar:na]
>       at com.apigee.keymanagement.util.ResourceUtil.getConsumerDetailsForApiKey(ResourceUtil.java:596)
> [keymanagement-1.0.0.jar:na]
>       at com.apigee.keymanagement.service.OAuth2RuntimeServiceImpl.getConsumerForApiKey(OAuth2RuntimeServiceImpl.java:81)
> [keymanagement-1.0.0.jar:na]
>       at com.apigee.oauth.v2.connectors.LocalOAuthServiceConnector.getClientAttributesForApiKey(LocalOAuthServiceConnector.java:173)
> [oauthV2-1.0.0.jar:na]
>       at com.apigee.oauth.v2.OAuthServiceImpl.getClientAttributesForApiKey(OAuthServiceImpl.java:506)
> [oauthV2-1.0.0.jar:na]
>       at com.apigee.steps.oauth.v2.OAuthStepExecution.execute(OAuthStepExecution.java:401)
> [oauthV2-1.0.0.jar:na]
>       at com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:97)
> [message-processor-1.0.0.jar:na]
>       at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:69)
> [message-flow-1.0.0.jar:na]
>       at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:51)
> [message-flow-1.0.0.jar:na]
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [na:1.6.0_32]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_32]
>       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [na:1.6.0_32]
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [na:1.6.0_32]
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_32]
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [na:1.6.0_32]
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [na:1.6.0_32]
>       at java.lang.Thread.run(Thread.java:662) [na:1.6.0_32]
>     Caused by: java.net.SocketException: Broken pipe
>       at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_32]
>       at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> ~[na:1.6.0_32]
>       at java.net.SocketOutputStream.write(SocketOutputStream.java:136) ~[na:1.6.0_32]
>       at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
> ~[libthrift-0.7.0.jar:0.7.0]
>       ... 31 common frames omitted
>     2014-04-01 14:29:20,126 org: env: Apigee-Main-36 ERROR m.p.c.c.HConnectionManager - HConnectionManager.markHostAsDown() :
> MARK HOST AS DOWN TRIGGERED for host 10.49.192.52(10.49.192.52):9160
>     2014-04-01 14:29:20,126 org: env: Apigee-Main-36 ERROR m.p.c.c.HConnectionManager - HConnectionManager.markHostAsDown() :
> Pool state on shutdown:
> <ConcurrentCassandraClientPoolByHost>:{10.49.192.52(10.49.192.52):9160};
> IsActive?: true; Active: 1; Blocked: 0; Idle: 2; NumBeforeExhausted: 9
>     2014-04-01 14:29:20,127 org: env: Apigee-Main-36 ERROR m.p.c.c.c.HThriftClient - HThriftClient.close() : Could not flush
> transport (to be expected if the pool is shutting down) in close for
> client: CassandraClient<10.49.192.52:9160-828>
>     org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection timed out
>       at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
> ~[libthrift-0.7.0.jar:0.7.0]
>       at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
> ~[libthrift-0.7.0.jar:0.7.0]
>       at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:125)
> [hector-core-1.1-3.jar:na]

2 个答案:

答案 0 :(得分：0)

当cassandra加载时会发生这种情况，并且会自行标记。

在此过程中，它将关闭池中的连接。

请重新启动消息处理器以重新建立连接。

让我知道它是怎么回事。

此致 Jagjyot。

答案 1 :(得分：0)

由于我是付费的Apigee客户，我也开了个案......

最初，他们不确定是否存在保持活动功能或连接TTL会强制丢弃/重新建立连接

这是我得到的：

  1. 在路由器/消息处理器节点上设置以下内容：/ proc / sys / net / ipv4 / tcp_keepalive_time至1800秒（30分钟）。

    2. 要做到这一点： echo 1800＆gt; / proc / sys / net / ipv4 / tcp_keepalive_time

    请注意：此更改不会在重新启动后保留，因此您需要编辑/etc/sysctl.conf文件并将其放在那里。

    然后执行命令：

    sysctl -p

    使这些值从该文件加载。

    您可以使用以下内容检查值是否已更新

    sysctl net.ipv4.tcp_keepalive_time

    1. 重新启动消息处理器。

      2. 因此，已经实施的修复是消息处理器中Hector客户端中的保持活动探测。

      探针根据tcp_keepalive_time OS设置中设置的间隔执行保持活动ping。 因此，将此设置为30分钟的原因是基于您的防火墙设置，空闲时间为3600秒。

      保持活动探测需要比防火墙的空闲超时更快地发生，以便它保持连接处于已建立状态。

相关问题
最新问题