Titan + DynamoDB Local非常慢（30个顶点的8s提交）

Question

我正在使用Titan + DynamoDB Local作为存储后端开发应用程序。以前我使用Berkeley作为存储后端，具有很好的性能。自从切换以来，我甚至无法使用该应用程序，因为一切都需要很长时间。

一个简单的测试用例添加30个顶点并提交更改;这就是我得到的：

//Berkeley Backend
Committing took 0.092 seconds.

//DynamoDB Local Backend    
Committing took 8.684 seconds.

后续提交需要同样长的时间。我知道dynamodb不会像伯克利那样快，因为它只是一个用于开发的包装SQL数据库，但它必须比这更快。我只能猜测我的配置有些不对劲，另一方面又花了straight from their repository。在内存中运行发电机只会略微提高性能。

我已经尝试过增加容量读/写单元而没有运气。有没有人在本地运行适度快速的发电机版本？

这是我目前的配置：

#general Titan configuration
gremlin.graph=com.thinkaurelius.titan.core.TitanFactory
storage.setup-wait=60000
storage.buffer-size=1024
# Metrics configuration - http://s3.thinkaurelius.com/docs/titan/1.0.0/titan-config-ref.html#_metrics
#metrics.enabled=true
#metrics.prefix=t
# Required; specify logging interval in milliseconds
#metrics.csv.interval=500
#metrics.csv.directory=metrics
# Turn off titan retries as we batch and have our own exponential backoff strategy.
storage.write-time=1 ms
storage.read-time=1 ms
storage.backend=com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager

#Amazon DynamoDB Storage Backend for Titan configuration
storage.dynamodb.force-consistent-read=true
# should be the graph name rexster/graphs/graph/graph-name
storage.dynamodb.prefix=v100
storage.dynamodb.metrics-prefix=d
storage.dynamodb.enable-parallel-scans=false
storage.dynamodb.max-self-throttled-retries=60
storage.dynamodb.control-plane-rate=10

# DynamoDB client configuration: credentials
storage.dynamodb.client.credentials.class-name=com.amazonaws.auth.BasicAWSCredentials
storage.dynamodb.client.credentials.constructor-args=access,secret

# DynamoDB client configuration: endpoint (Below, set to DynamoDB Local as invoked by mvn test -Pstart-dynamodb-local).
# You can change the endpoint to point to Production DynamoDB regions.)
storage.dynamodb.client.endpoint=http://localhost:1234

# max http connections - not recommended to use more than 250 connections in DynamoDB Local
storage.dynamodb.client.connection-max=250
# turn off sdk retries
storage.dynamodb.client.retry-error-max=0

# DynamoDB client configuration: thread pool
storage.dynamodb.client.executor.core-pool-size=25
# Do not need more threads in thread pool than the number of http connections
storage.dynamodb.client.executor.max-pool-size=250
storage.dynamodb.client.executor.keep-alive=60000
storage.dynamodb.client.executor.max-concurrent-operations=1
# should be at least as large as the storage.buffer-size
storage.dynamodb.client.executor.max-queue-length=1024

# 750 r/w CU result in provisioning the maximum equal numbers read and write Capacity Units that can
# be set on one table before it is split into two or more partitions for IOPS. If you will have more than one Rexster server
# accessing the same graph, you should set the read-rate and write-rate properties to values commensurately lower than the
# read and write capacity of the backend tables.

storage.dynamodb.stores.edgestore.capacity-read=100
storage.dynamodb.stores.edgestore.capacity-write=100
storage.dynamodb.stores.edgestore.read-rate=100
storage.dynamodb.stores.edgestore.write-rate=100
storage.dynamodb.stores.edgestore.scan-limit=10000

storage.dynamodb.stores.graphindex.capacity-read=100
storage.dynamodb.stores.graphindex.capacity-write=100
storage.dynamodb.stores.graphindex.read-rate=100
storage.dynamodb.stores.graphindex.write-rate=100
storage.dynamodb.stores.graphindex.scan-limit=10000

storage.dynamodb.stores.systemlog.capacity-read=10
storage.dynamodb.stores.systemlog.capacity-write=10
storage.dynamodb.stores.systemlog.read-rate=10
storage.dynamodb.stores.systemlog.write-rate=10
storage.dynamodb.stores.systemlog.scan-limit=10000

storage.dynamodb.stores.titan_ids.capacity-read=10
storage.dynamodb.stores.titan_ids.capacity-write=10
storage.dynamodb.stores.titan_ids.read-rate=10
storage.dynamodb.stores.titan_ids.write-rate=10
storage.dynamodb.stores.titan_ids.scan-limit=10000

storage.dynamodb.stores.system_properties.capacity-read=10
storage.dynamodb.stores.system_properties.capacity-write=10
storage.dynamodb.stores.system_properties.read-rate=10
storage.dynamodb.stores.system_properties.write-rate=10
storage.dynamodb.stores.system_properties.scan-limit=10000

storage.dynamodb.stores.txlog.capacity-read=10
storage.dynamodb.stores.txlog.capacity-write=10
storage.dynamodb.stores.txlog.read-rate=10
storage.dynamodb.stores.txlog.write-rate=10
storage.dynamodb.stores.txlog.scan-limit=10000

# elasticsearch config that is required to run GraphOfTheGods
index.search.backend=elasticsearch
index.search.directory=/tmp/searchindex
index.search.elasticsearch.client-only=false
index.search.elasticsearch.local-mode=true
index.search.elasticsearch.interface=NODE

测试代码：

private long start;

    @Test
    public void performanceTest() {
        Graph graph = GraphFactory.getDefault();

        for (int i=1; i<=30; i++) {
            graph.addVertex("myLabel");
        }

        startStopwatch();
        graph.tx().commit();
        stopStopwatch("Committing");
    }

    private void startStopwatch() {
        start = System.currentTimeMillis();
    }

    private void stopStopwatch(String opName) {
        System.out.println(opName + " took " + (System.currentTimeMillis() - start) / 1000.0 + " seconds.");
    }

版本

• Titan 1.0.0
• DynamoDB Storage Backend 1.0.2
• DynamoDB Local 1.11.86
• Berkeley 1.0.0

相关

• DynamoDB Local Terrible Performance
• Terrible Write Performance with DynamoDB Local

Answer 1

我created an issue on GitHub来跟踪您感知的延迟的再现。我在运行代码时仅在命令行上测量84 ms的提交时间，即使我包含所有设置和静态Java初始化代码，端到端测试也只需要5秒。请试用master branch。

final Graph graph = JanusGraphFactory.open(TestGraphUtil.instance.createTestGraphConfig(BackendDataModel.MULTI));
IntStream.of(30).forEach(i -> graph.addVertex(LABEL));
Stopwatch watch = Stopwatch.createStarted();
graph.tx().commit();
System.out.println("Committing took " + watch.stop().elapsed(TimeUnit.MILLISECONDS) + " ms");
TestGraphUtil.instance.cleanUpTables();

DynamoDB Local是一种测试工具，您不应期望在使用DynamoDB服务时看到相同的低延迟。话虽如此，当您使用inMemory选项时，您将获得DynamoDB Local的最佳性能。