我们在其中一项服务中使用Titan Distributed Graph Database。我们有一个CassandraDB作为存储后端的工作配置。
现在我们正在使用AWS DynamoDB存储后端运行该服务。如果我们使用https://github.com/awslabs/dynamodb-titan-storage-backend以本地模式运行它,一切正常。但是,在迁移到云中的AWS DynamoDB后,我们发现了一些不确定性问题。
我们的单元测试运行在空数据库上,生成测试数据,执行一些CRUD,然后清理所有内容。测试通过Cassandra。 它们也传递本地DynamoDB。不幸的是,它们往往在本地DynamoDB和云中的AWS DynamoDB上失败。通常在发生故障之前需要几次运行。在这样的失败之后,我们可以在数据库中找到一些鬼顶点。看起来索引与图表不同步。在结果中,我们可以在使用索引查询图形而不使用它时看到不同的结果。
日志中也有例外。最有趣的是: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException:在parallelMutate期间被中断:
13:19:12 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:12 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:13 [ERROR] [c.h.c.c.h.HystrixDiagnosticLogger] [] HYSTRIX: command category-v1.createCategory failed (com.hybris.caas.category.command.CreateCategoryCommand@7a77b317); execution events: [TIMEOUT, FALLBACK_MISSING] java.util.concurrent.TimeoutException: null at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:600) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:580) ~[hystrix-core-1.5.1.jar:1.5.1] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:955) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:972) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) [hystrix-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] 13:19:13 [ERROR] [c.t.t.g.database.StandardTitanGraph] [] Could not commit transaction [9] due to storage exception in commit com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:na] at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) [classes/:na] at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1] at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na] ... 40 common frames omitted Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na] ... 41 common frames omitted 13:19:13 [INFO ] [c.h.c.c.c.w.m.TimeoutExceptionMapper] [] Mapping TimeoutException java.util.concurrent.TimeoutException: null at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:600) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:580) ~[hystrix-core-1.5.1.jar:1.5.1] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:955) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) ~[hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) ~[service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:972) ~[hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) ~[hystrix-core-1.5.1.jar:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_102] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102] 13:19:13 [ERROR] [c.t.t.g.database.StandardTitanGraph] [] Could not commit transaction [9] due to exception com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:na] at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) [classes/:na] at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1] at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na] ... 40 common frames omitted Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na] ... 41 common frames omitted 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doRollback() 13:19:13 [ERROR] [c.h.c.c.t.TitanTransactionManager] [] Commit exception overridden by rollback exception com.thinkaurelius.titan.core.TitanException: Could not commit transaction due to exception during persistence at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1363) ~[titan-core-1.0.0.jar:na] at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) ~[classes/:na] at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE] at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na] at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1] at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1] at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1] at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1] at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0] at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1] at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) ~[titan-core-1.0.0.jar:na] ... 34 common frames omitted Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na] ... 40 common frames omitted Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na] at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na] ... 41 common frames omitted 13:19:13 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/CategoryCreation.json 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:14 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/AssignmentCreation.json 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.r.impl.utils.ReferenceVertex] [] Creating vertex for com.hybris.caas.category.model.Reference@2accc739[id=NEW,type=product,url=product-service.dev.cf.hybris.com/NEW] 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:14 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/AssignmentCreation.json 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doRollback() 13:19:14 [INFO ] [c.h.c.c.c.DeleteCategoriesCommand] [] [AUDIT [ remove_all_categories ]] [user=User[name=,roles=[hybris.category_delete_all]], tenant=womdev1s] Delete all categories. 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:14 [INFO ] [c.h.c.c.r.i.GraphCategoryRepository] [] Deleting all categories in tenant 'womdev1s' 13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:16 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/CategoryCreation.json 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:16 [INFO ] [c.h.c.c.r.impl.utils.CategoryVertex] [] Creating root category vertex for tenant 'womdev1s' 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin() 13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit() ......
我知道我们应该期待一些问题,因为"最终的一致性"。 CassandraDB和DynamoDB似乎最终都是一致的,但在我们的例子中,行为明显不同。我们也在使用" storage.dynamodb.force-consistent-read = true"标志。
有没有人试图面对这样的问题?
答案 0 :(得分:0)
引入Titan/JanusGraph based locking(practical example)可能会纠正您遇到的索引一致性问题。无论后端选择如何,鬼顶点始终是Titan / JanusGraph中的主题,并且该前端的指导是使用OLAP作业扫描图形并消除不一致。