Titan / DynamoDb不会在提交时释放所有获取的锁（通过gremlin）

时间：2017-02-07 13:03:40

标签： titan gremlin tinkerpop3

好吧，我意识到这听起来不太可能，而且我已经准备好对这一个进行击落，但是这里有......

我有一个针对titanDB和dynamoDB（本地）运行的gremlin服务器。我正在运行一些由于

而导致失败的单元测试 
tx 0x705eafda280e already locked key-column (  8-  0-  0-  0-  0-  0-  0-128, 80-160) when tx 0x70629e1d56bf tried to lock

我在gremlin客户端控制台中运行以下命令，对着干净，完全空的DB（在使用docker镜像的测试运行之间重新创建）。这项工作的目的是支持数据库升级脚本。最初的实际步骤比下面的更完整，但这是重现问题的最小步骤。

(Connect to local 'remote')
:remote connect tinkerpop.server conf/remote.yaml

(Add a unique constraint on a 'databaseMetadata' label which has a single 'version' property)
:> mgmt = graph.openManagement();if (!mgmt.getGraphIndex("bydatabaseMetadataversion")) {graph.tx().rollback();int size = graph.getOpenTransactions().size();for (i = 0; i < size; i++) { try { graph.getOpenTransactions().getAt(0).rollback();} catch(Throwable ex) { }; }; mgmt = graph.openManagement();propertyKey = (!mgmt.containsPropertyKey("version")) ? mgmt.makePropertyKey("version").dataType(String.class).cardinality(Cardinality.SINGLE).make():mgmt.getPropertyKey("version");labelObj = (!mgmt.containsVertexLabel("databaseMetadata")) ? mgmt.makeVertexLabel("databaseMetadata").make():mgmt.getVertexLabel("databaseMetadata");index = mgmt.buildIndex("bydatabaseMetadataversion", Vertex.class).addKey(propertyKey).unique().indexOnly(labelObj).buildCompositeIndex();mgmt.setConsistency(propertyKey, ConsistencyModifier.LOCK);mgmt.setConsistency(index, ConsistencyModifier.LOCK);mgmt.commit();mgmt = graph.openManagement();index = mgmt.getGraphIndex("bydatabaseMetadataversion");propertyKey = mgmt.getPropertyKey("version");if (index.getIndexStatus(propertyKey) == SchemaStatus.INSTALLED) {mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();}; mgmt.commit();mgmt = graph.openManagement();index = mgmt.getGraphIndex("bydatabaseMetadataversion");propertyKey = mgmt.getPropertyKey("version");if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {mgmt.commit();mgmt = graph.openManagement();mgmt.updateIndex(mgmt.getGraphIndex("bydatabaseMetadataversion"), SchemaAction.ENABLE_INDEX).get();mgmt.commit();mgmt = graph.openManagement();mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();}; mgmt.commit();} else {index = mgmt.getGraphIndex("bydatabaseMetadataversion");propertyKey = mgmt.getPropertyKey("version");if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();}; mgmt.commit();};

(Add the metadata vertex with initial version '0.0.1')
:> graph.addVertex(label, "databaseMetadata").property("version", "0.0.1");graph.tx().commit();

(Update the metadata vertex with the next version - 0.0.2)
:> g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").next();g.tx().commit();

(THIS FAILS - Update the metadata vertex with the next version - 0.0.3)
:> g.V().hasLabel("databaseMetadata").has("version", "0.0.2").property("version", "0.0.3").next();g.tx().commit();
tx 0x705eafda280e already locked key-column (  8-  0-  0-  0-  0-  0-  0-128, 80-160) when tx 0x70629e1d56bf tried to lock

之前我查看了titan-dynamodb源代码，我发现事务的提交/回滚等都已记录，所以我更改了日志级别以获取更多信息（可用完整的日志文件）。

当0.0.1 - >执行了0.0.2更新，获得了以下锁：

[33mtitan_server_1  |[0m 120479 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore  - acquiring lock on (  8-  0-  0-  0-  0-  0-  0-128, 80-160) at 123552624951495
[33mtitan_server_1  |[0m 120489 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore  - acquiring lock on (  6-137-160- 48- 46- 48- 46-177,  0) at 123552635424334
[33mtitan_server_1  |[0m 120489 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore  - acquiring lock on (  6-137-160- 48- 46- 48- 46-178,  0) at 123552635704705

当提交该事务时，只有两个锁被释放。

[33mtitan_server_1  |[0m 120722 [gremlin-server-exec-3] DEBUG com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreTransaction  - commit id:0x705eafda280e
[33mtitan_server_1  |[0m 120722 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore  - Expiring (  6-137-160- 48- 46- 48- 46-177,  0) in tx 0x705eafda280e because of EXPLICIT
[33mtitan_server_1  |[0m 120722 [gremlin-server-exec-3] TRACE com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore  - Expiring (  6-137-160- 48- 46- 48- 46-178,  0) in tx 0x705eafda280e because of EXPLICIT
[33mtitan_server_1  |[0m 120722 [gremlin-server-exec-3] DEBUG org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Preparing to iterate results from - RequestMessage{, requestId=09f27811-dcc3-4e53-a749-22828d34997f, op='eval', processor='', args={gremlin=g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").next();g.tx().commit();, batchSize=64}} - in thread [gremlin-server-exec-3]

剩余的锁定会在几分钟后到期，但同时每个其他更新都会因报告而失败。

那么，为什么锁不会被删除？我怀疑它与创建的唯一索引有关，所以我要么设置索引错误（很可能）或者这是一个错误。

为便于消费，（略微缩短的）索引设置如下：

mgmt = graph.openManagement()
propertyKey = (!mgmt.containsPropertyKey("version")) ? mgmt.makePropertyKey("version").dataType(String.class).cardinality(Cardinality.SINGLE).make():mgmt.getPropertyKey("version")
labelObj = (!mgmt.containsVertexLabel("databaseMetadata")) ? mgmt.makeVertexLabel("databaseMetadata").make():mgmt.getVertexLabel("databaseMetadata")
index = mgmt.buildIndex("bydatabaseMetadataversion", Vertex.class).addKey(propertyKey).unique().indexOnly(labelObj).buildCompositeIndex()
mgmt.setConsistency(propertyKey, ConsistencyModifier.LOCK)
mgmt.setConsistency(index, ConsistencyModifier.LOCK)
mgmt.commit()
mgmt = graph.openManagement()
index = mgmt.getGraphIndex("bydatabaseMetadataversion")
propertyKey = mgmt.getPropertyKey("version")
if (index.getIndexStatus(propertyKey) == SchemaStatus.INSTALLED) {
  mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call()
}
mgmt.commit()
mgmt = graph.openManagement()
index = mgmt.getGraphIndex("bydatabaseMetadataversion")
propertyKey = mgmt.getPropertyKey("version")
if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {
  mgmt.commit()
  mgmt = graph.openManagement()
  mgmt.updateIndex(mgmt.getGraphIndex("bydatabaseMetadataversion"), SchemaAction.ENABLE_INDEX).get()
  mgmt.commit()
  mgmt = graph.openManagement()
  mgmt.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call()
}
mgmt.commit()

我知道这是一个很长的问题描述，但感谢任何帮助！

（我还应该说，我针对本地和基于云的dynamoDb实例尝试了这一点，并且两者都有相同的问题，所以回到本地并开启了日志记录。）

我正在使用dynamo-titan on github中设置的titan 1.0.0和tinkerpop 3。

2 个答案:

答案 0 :(得分：2)

F.y.i。我使用Berkeley存储后端在Java中运行了上述所有代码。

TitanGraph graph = ...;
TitanManagement mgmt = graph.openManagement();
PropertyKey propertyKey = (!mgmt.containsPropertyKey("version"))
        ? mgmt.makePropertyKey("version").dataType(String.class).cardinality(Cardinality.SINGLE).make()
        : mgmt.getPropertyKey("version");
VertexLabel labelObj = (!mgmt.containsVertexLabel("databaseMetadata"))
        ? mgmt.makeVertexLabel("databaseMetadata").make() 
        : mgmt.getVertexLabel("databaseMetadata");
TitanGraphIndex index = mgmt.buildIndex("bydatabaseMetadataversion", Vertex.class).addKey(propertyKey).unique()
        .indexOnly(labelObj).buildCompositeIndex();
mgmt.setConsistency(propertyKey, ConsistencyModifier.LOCK);
mgmt.setConsistency(index, ConsistencyModifier.LOCK);
mgmt.commit();
mgmt = graph.openManagement();
index = mgmt.getGraphIndex("bydatabaseMetadataversion");
propertyKey = mgmt.getPropertyKey("version");
if (index.getIndexStatus(propertyKey) == SchemaStatus.INSTALLED) {
    try {
        ManagementSystem.awaitGraphIndexStatus(graph,"bydatabaseMetadataversion").status(SchemaStatus.REGISTERED).timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}
mgmt.commit();
mgmt = graph.openManagement();
index = mgmt.getGraphIndex("bydatabaseMetadataversion");
propertyKey = mgmt.getPropertyKey("version");
if (index.getIndexStatus(propertyKey) != SchemaStatus.ENABLED) {
    mgmt.commit();
    mgmt = graph.openManagement();
    try {
        mgmt.updateIndex(mgmt.getGraphIndex("bydatabaseMetadataversion"), SchemaAction.ENABLE_INDEX).get();
    } catch (InterruptedException | ExecutionException e) {
        e.printStackTrace();
    }
    mgmt.commit();
    mgmt = graph.openManagement();
    try {
        ManagementSystem.awaitGraphIndexStatus(graph, "bydatabaseMetadataversion").status(SchemaStatus.ENABLED)
                        .timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}
mgmt.commit();

然后对图表进行操作; 

GraphTraversalSource g = graph.traversal();

graph.addVertex("databaseMetadata").property("version", "0.0.1");
graph.tx().commit();

g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").iterate();
g.tx().commit();

g.V().hasLabel("databaseMetadata").has("version", "0.0.1").property("version", "0.0.2").iterate();
g.tx().commit();

g.V().hasLabel("databaseMetadata").has("version", "0.0.2").property("version", "0.0.3").iterate();
g.tx().commit();

g.V().hasLabel("databaseMetadata").has("version").properties("version").forEachRemaining(prop -> {
    System.out.println("Version: " + prop.value());
});

结果是：

Version: 0.0.3

遗憾的是，查询的iterate()更改仅适用于Java。您的脚本应该按原样运行。由于我的实验结果，我强烈怀疑DynamoDB后端造成了麻烦。

答案 1 :(得分：2)

我得到了repro并找到了你的问题。基本上，LRU缓存从storage.lock.expiry-time config中拉出其到期时间。默认值为5分钟，因此如果您尝试在5分钟之前进行更改，是的，则AbstractDynamoDBStore.keyColumnLocalLocks LRU缓存将不允许您进行第二次更改。通过在进行第二次更改之前减少到期时间和Thread.sleep（），您允许第二次更改再次声明锁定并成功。

//default lock expiry time is 300*1000 ms = 5 minutes. Set to 100ms.
config.setProperty("storage.lock.expiry-time", 100);
相关问题
最新问题