Kaa集群引起的问题

时间:2016-12-12 08:37:43

标签: kaa

我在the kaa Cluster setup guide document.之后部署了3个Kaa集群。数据库是Cassandra和MariaDB。当我运行C客户端时(我做了一个代码更改,每个客户端都可以随机登录三个kaa服务器)。服务器日志文件中报告了很多问题。我将在Kaa Cluster中上传one Server logs。我总结了一下,有7个错误。当C客户端首先登录kaa服务器时,会出现第一个错误。在我停止C客户端并再次运行后,错误没有出现。我认为第二个错误是由数据库集群引起的,也许是数据库一致性。因为我限制使用连接到两个Cassandra的10 MB HUB,所以经常会出现错误。我试图弄清楚其他错误并且现在不知道。

1 -------------------

2016-12-09 00:00:43,950 [EPS-endpoint-dispatcher-597] ERROR o.k.k.s.c.n.c.d.AbstractVersionableCassandraDao - [class org.kaaproject.kaa.server.common.nosql.cassandra.dao.model.CassandraEndpointProfile] Can't update entity with version 2. Entity already changed!
2016-12-09 00:00:43,952 [EPS-endpoint-dispatcher-597] WARN  o.k.k.s.c.d.s.EndpointServiceImpl - Optimistic lock detected in endpoint profile 
org.kaaproject.kaa.server.common.dao.exception.KaaOptimisticLockingFailureException: Can't update entity with version 2. Entity already changed!
    at org.kaaproject.kaa.server.common.nosql.cassandra.dao.AbstractVersionableCassandraDao.updateLocked(AbstractVersionableCassandraDao.java:100) ~[cassandra-dao-0.10.0.jar:na]
    at org.kaaproject.kaa.server.common.nosql.cassandra.dao.AbstractVersionableCassandraDao.save(AbstractVersionableCassandraDao.java:59) ~[cassandra-dao-0.10.0.jar:na]
    at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.updateProfile(EndpointProfileCassandraDao.java:168) ~[cassandra-dao-0.10.0.jar:na]
    at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:129) ~[cassandra-dao-0.10.0.jar:na]
    at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:121) ~[cassandra-dao-0.10.0.jar:na]
    at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:90) ~[cassandra-dao-0.10.0.jar:na]
    at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at com.sun.proxy.$Proxy64.save(Unknown Source) ~[na:na]
    at org.kaaproject.kaa.server.common.dao.service.EndpointServiceImpl.saveEndpointProfile(EndpointServiceImpl.java:382) ~[dao-0.10.0.jar:na]
    at org.kaaproject.kaa.server.common.dao.service.EndpointServiceImpl.attachEndpointToUser(EndpointServiceImpl.java:421) ~[dao-0.10.0.jar:na]
    at sun.reflect.GeneratedMethodAccessor180.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:121) [spring-retry-1.1.2.RELEASE.jar:na]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at com.sun.proxy.$Proxy70.attachEndpointToUser(Unknown Source) [na:na]
    at org.kaaproject.kaa.server.operations.service.user.DefaultEndpointUserService.attachEndpointToUser(DefaultEndpointUserService.java:93) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.attachEndpointToUser(DefaultOperationsService.java:788) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.processUserVerificationMessage(LocalEndpointActorMessageProcessor.java:1025) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.processUserVerificationMessage(LocalEndpointActor.java:123) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.onReceive(LocalEndpointActor.java:88) [kaa-node-0.10.0.jar:na]
    at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.Actor$class.aroundReceive(Actor.scala:480) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.1.jar:na]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.1.jar:na]
    at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.1.jar:na]
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.1.jar:na]
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]





2 -------------------


2016-12-09 00:17:12,473 [EPS-endpoint-dispatcher-586] ERROR o.k.k.s.o.s.a.a.c.e.l.LocalEndpointActorMessageProcessor - [R18PpFaouxXSrHQVbOfyTtiGKBo=][cd72eab3ccec4135a6ef4f553add7bb6] processEndpointRequest
java.security.InvalidParameterException: Cache key can't be null
    at org.kaaproject.kaa.server.operations.service.cache.concurrent.CacheTemporaryMemorizer.compute(CacheTemporaryMemorizer.java:77) ~[kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.cache.concurrent.ConcurrentCacheService.getConfByHash(ConcurrentCacheService.java:581) ~[kaa-node-0.10.0.jar:na]
    at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:320) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:353) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:302) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
    at com.sun.proxy.$Proxy96.getConfByHash(Unknown Source) ~[na:na]
    at org.kaaproject.kaa.server.operations.service.delta.DefaultDeltaService.getDelta(DefaultDeltaService.java:155) ~[kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.calculateConfigurationDelta(DefaultOperationsService.java:675) ~[kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.syncConfiguration(DefaultOperationsService.java:426) ~[kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.sync(LocalEndpointActorMessageProcessor.java:394) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.sync(LocalEndpointActorMessageProcessor.java:310) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.processEndpointSync(LocalEndpointActorMessageProcessor.java:135) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.processEndpointSync(LocalEndpointActor.java:131) [kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.onReceive(LocalEndpointActor.java:78) [kaa-node-0.10.0.jar:na]
    at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.Actor$class.aroundReceive(Actor.scala:480) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.1.jar:na]
    at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.1.jar:na]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.1.jar:na]
    at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.1.jar:na]
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.1.jar:na]
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]


3  -----------------



2016-12-09 00:32:35,091 [EPS-core-dispatcher-1174] ERROR o.k.k.s.o.s.a.a.s.SupervisionStrategyFactory - Supervisor strategy got exception: null
java.lang.NullPointerException: null
2016-12-09 00:32:35,092 [EPS-akka.actor.default-dispatcher-4] ERROR akka.actor.OneForOneStrategy - null
java.lang.NullPointerException: null
2016-12-09 00:32:35,097 [EPS-endpoint-dispatcher-586] INFO  o.k.k.s.o.s.u.DefaultEndpointUserService - Can't find listeners for unassigned endpoint!




4 --------------------




2016-12-09 01:15:46,710 [pool-22-thread-1] ERROR o.k.k.s.thrift.NeighborConnection - EventWorker [192.168.10.14:9090:OPERATIONS_SERVICE:d9c71186-e45e-4c33-88b0-0ad5f26c6d17] error sending event messages pack. 
com.twitter.common.thrift.TResourceExhaustedException: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use
    at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:91) ~[thrift-0.0.64.jar:na]
    at com.twitter.common.thrift.Thrift$1.invoke(Thrift.java:363) ~[thrift-0.0.64.jar:na]
    at com.sun.proxy.$Proxy141.sendMessages(Unknown Source) ~[na:na]
    at org.kaaproject.kaa.server.operations.service.event.DefaultEventService$MessageTemplate.process(DefaultEventService.java:727) ~[kaa-node-0.10.0.jar:na]
    at org.kaaproject.kaa.server.thrift.NeighborConnection$EventWorker.run(NeighborConnection.java:322) ~[server-shared-0.10.0.jar:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use
    at com.twitter.common.thrift.ThriftFactory$ManagedThreadPool$1.rejectedExecution(ThriftFactory.java:237) ~[thrift-0.0.64.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) [na:1.8.0_111]
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[na:1.8.0_111]
    at com.twitter.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:66) ~[util-0.0.88.jar:na]
    at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:71) ~[thrift-0.0.64.jar:na]
    ... 9 common frames omitted




5 ---------------------



2016-12-09 02:15:50,233 [EPS-endpoint-dispatcher-598] ERROR o.k.k.s.o.s.a.a.c.e.l.LocalEndpointActorMessageProcessor - [Nje8C3i2b6K7ukh7tP8G/pDzdrE=][3f5e75f03bd34c30be183807486176ba] processEndpointRequest
java.lang.NullPointerException: null




6 -------------------------


2016-12-09 00:53:05,649 [EPS-core-dispatcher-1117] ERROR o.k.k.s.o.s.a.a.s.SupervisionStrategyFactory - Supervisor strategy got exception: null
java.lang.NullPointerException: null
2016-12-09 00:53:05,650 [EPS-akka.actor.default-dispatcher-4] ERROR akka.actor.OneForOneStrategy - null
java.lang.NullPointerException: null


7 ------------------------


2016-12-09 09:20:33,680 [qtp1981719029-883] ERROR o.k.k.s.admin.servlet.SdkServlet - Unexpected error in SdkServlet.doGet: 
org.eclipse.jetty.io.EofException: null
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:402) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:111) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:577) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:548) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:488) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) ~[jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:606) [jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:535) [jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_111]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_111]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_111]
    at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_111]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_111]
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    ... 9 common frames omitted

1 个答案:

答案 0 :(得分:0)

似乎这与KAA-1643错误有关。它已经修复,但尚未发布(将在0.10.1中)。您可以尝试使用该修复程序构建最新版本。