neo4j ::如何在嵌入式模型中设置短路径查询超时属性?

时间:2014-11-24 02:25:56

标签: neo4j timeout

我使用的是neo4j 1.9.4嵌入式模型。执行短路查询(50万个节点和600万个关系)时花了太长时间。我阅读了手册并找到了两个属性:

1. conf/neo4j-server.properties
org.neo4j.server.webserver.limit.executiontime=1000
2. conf/neo4j.properties
execution_guard_enabled=true

但问题是,似乎没有' neo4j-server.properties'嵌入式模型中的文件。如何设置?或者“超时”'嵌入式模型不支持属性吗?

请帮忙。非常感谢!

代码:

PathFinder<Path> finder = GraphAlgoFactory.shortestPath(Traversal
                        .expanderForTypes(RelationshipTypes.r01, Direction.BOTH)
                        .add(RelationshipTypes.r02, Direction.BOTH), 10);
paths = finder.findAllPaths(node1, node2);
for (Path shortestPath : paths) {
    List<NodeBean> nodeList = new ArrayList<NodeBean>();
    List<String> rList = new ArrayList<String>();
    ... ...
}

message.log的一部分:

    2014-11-24 02:06:42.107+0000 INFO  [org.neo4j]: --- INITIALIZED diagnostics END ---
    2014-11-24 02:06:42.217+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.relationshiptypestore.db.names non clean shutdown detected
    2014-11-24 02:06:42.217+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.relationshiptypestore.db.names] brickCount=0 brickSize=0b mappedMem=0b (storeSize=114b)
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.relationshiptypestore.db non clean shutdown detected
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.relationshiptypestore.db] brickCount=0 brickSize=0b mappedMem=0b (storeSize=10b)
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.strings non clean shutdown detected
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.propertystore.db.strings] brickCount=10 brickSize=20864b mappedMem=20971520b (storeSize=209664b)
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.index.keys non clean shutdown detected
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.propertystore.db.index.keys] brickCount=50 brickSize=20938b mappedMem=20971520b (storeSize=1048572b)
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.index non clean shutdown detected
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.propertystore.db.index] brickCount=261 brickSize=20970b mappedMem=20971520b (storeSize=5473170b)
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.arrays non clean shutdown detected
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.propertystore.db.arrays] brickCount=0 brickSize=20864b mappedMem=20971520b (storeSize=128b)
    2014-11-24 02:06:42.232+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db non clean shutdown detected
    2014-11-24 02:06:42.248+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.propertystore.db] brickCount=23513 brickSize=104837b mappedMem=104857600b (storeSize=2465619255b)
    2014-11-24 02:06:42.248+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.relationshipstore.db non clean shutdown detected
    2014-11-24 02:06:42.248+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.relationshipstore.db] brickCount=100000 brickSize=221826b mappedMem=104857600b (storeSize=22183075860b)
    2014-11-24 02:06:42.248+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.nodestore.db non clean shutdown detected
    2014-11-24 02:06:42.248+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore.nodestore.db] brickCount=4733 brickSize=104877b mappedMem=104857600b (storeSize=496395405b)
    2014-11-24 02:06:42.248+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore non clean shutdown detected
    2014-11-24 02:06:42.248+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\neostore] brickCount=0 brickSize=0b mappedMem=0b (storeSize=54b)
    2014-11-24 02:06:42.263+0000 INFO  [org.neo4j]: Non clean shutdown detected on log [D:\FTP\graph1.db\nioneo_logical.log.1]. Recovery started ...
    2014-11-24 02:06:42.279+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\nioneo_logical.log.1] logVersion=4730 with committed tx=1072646
    2014-11-24 02:06:42.279+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\nioneo_logical.log.1] entries found=0 lastEntryPos=16
    2014-11-24 02:06:42.279+0000 INFO  [org.neo4j]: Opened logical log [D:\FTP\graph1.db\nioneo_logical.log.1] version=4730, lastTxId=-1 (recovered)
    2014-11-24 02:06:42.279+0000 INFO  [org.neo4j]: XaResourceManager[nioneo_logical.log] sorting 0 xids
    2014-11-24 02:06:42.279+0000 INFO  [org.neo4j]: XaResourceManager[nioneo_logical.log] checkRecoveryComplete 0 xids
    2014-11-24 02:06:42.419+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.relationshiptypestore.db.names rebuild id generator, highId=3 defragged count=0
    2014-11-24 02:06:42.934+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.index.keys rebuild id generator, highId=3 defragged count=0
    2014-11-24 02:06:46.351+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.index rebuild id generator, highId=2 defragged count=0
    2014-11-24 02:06:46.444+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.strings rebuild id generator, highId=1490 defragged count=0
    2014-11-24 02:06:46.538+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db.arrays rebuild id generator, highId=1 defragged count=0
    2014-11-24 02:06:46.694+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.propertystore.db rebuild id generator, highId=60134068 defragged count=0
    2014-11-24 02:06:46.943+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.relationshipstore.db rebuild id generator, highId=672185416 defragged count=0
    2014-11-24 02:06:47.255+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore.nodestore.db rebuild id generator, highId=55117690 defragged count=0
    2014-11-24 02:06:47.349+0000 INFO  [org.neo4j]: D:\FTP\graph1.db\neostore rebuild id generator, highId=6 defragged count=0
    2014-11-24 02:06:47.411+0000 INFO  [org.neo4j]: XaResourceManager[nioneo_logical.log] recovery completed.
    2014-11-24 02:06:47.411+0000 INFO  [org.neo4j]: Recovery on log [D:\FTP\graph1.db\nioneo_logical.log.1] completed.
    2014-11-24 02:06:47.411+0000 INFO  [org.neo4j]: TM opening log: D:\FTP\graph1.db\tm_tx_log.1
    2014-11-24 02:06:47.567+0000 INFO  [org.neo4j]: BackupServer communication server started and bound to /0.0.0.0:6362
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: Non clean shutdown detected on log [D:\FTP\graph1.db\index\lucene.log.1]. Recovery started ...
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\index\lucene.log.1] logVersion=252 with committed tx=5514
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: [D:\FTP\graph1.db\index\lucene.log.1] entries found=0 lastEntryPos=16
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: Opened logical log [D:\FTP\graph1.db\index\lucene.log.1] version=252, lastTxId=-1 (recovered)
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: XaResourceManager[lucene.log] sorting 0 xids
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: XaResourceManager[lucene.log] checkRecoveryComplete 0 xids
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: XaResourceManager[lucene.log] recovery completed.
    2014-11-24 02:06:47.599+0000 INFO  [org.neo4j]: Recovery on log [D:\FTP\graph1.db\index\lucene.log.1] completed.
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for KernelDiagnostics:Versions START ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: Graph Database: org.neo4j.kernel.EmbeddedGraphDatabase StoreId[time:1414568675229, id:-3029933565465970314, store version: 13561656364791302]
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: Kernel version: Neo4j - Graph Database Kernel 1.9.4
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: Neo4j component versions:
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]:   Neo4j - Graph Database Kernel 1.9.4
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for KernelDiagnostics:Versions END ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for NEO_STORE_VERSIONS START ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: Store versions:
        Store versions:
          NeoStore v0.A.0
          NodeStore v0.A.0
          RelationshipStore v0.A.0
          RelationshipTypeStore v0.A.0
          PropertyStore v0.A.0
          PropertyIndexStore v0.A.0
          StringPropertyStore v0.A.0
          ArrayPropertyStore v0.A.0
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for NEO_STORE_VERSIONS END ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for NEO_STORE_ID_USAGE START ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: Id usage:

         Id usage:
         NodeStore: used=55155045 high=55155044
         RelationshipStore: used=672214420 high=672214419
         RelationshipTypeStore: used=2 high=1
         PropertyStore: used=60137055 high=60137054
         PropertyIndexStore: used=608130 high=608129
         StringPropertyStore: used=1638 high=1637
         ArrayPropertyStore: used=1 high=0
        2014-11-24 02:06:47.614+0000 INFO  [org

    .neo4j]: --- STARTED diagnostics for NEO_STORE_ID_USAGE END ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for PERSISTENCE_WINDOW_POOL_STATS START ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for PERSISTENCE_WINDOW_POOL_STATS END ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: --- STARTED diagnostics for KernelDiagnostics:StoreFiles START ---
    2014-11-24 02:06:47.614+0000 INFO  [org.neo4j]: Disk space on partition (Total / Free / Free %): 316763721728 / 32676884480 / 10
    Storage files: (filename : modification date - size)
          active_tx_log: 2014-11-21T08:52:51+0800 - 11.00 B
          index:
            lucene:
              node:
                nodes:
                  segments.gen: 2014-11-21T12:28:14+0800 - 20.00 B
                  segments_4z: 2014-11-21T12:28:14+0800 - 5.25 kB
                  _111.fdt: 2014-11-21T12:28:53+0800 - 223.40 MB
                  _111.fdx: 2014-11-21T12:28:24+0800 - 55.69 MB
                  _111.fnm: 2014-11-21T12:28:24+0800 - 18.00 B
                  _111.frq: 2014-11-21T12:28:34+0800 - 53.68 MB
                  _111.nrm: 2014-11-21T12:28:35+0800 - 6.96 MB
                  _111.prx: 2014-11-21T12:28:37+0800 - 13.92 MB
                  _111.tii: 2014-11-21T12:28:38+0800 - 1.62 MB
                  _111.tis: 2014-11-21T12:28:58+0800 - 115.02 MB
                  _1ri.fdt: 2014-11-21T12:29:40+0800 - 263.73 MB
                  _1ri.fdx: 2014-11-21T12:29:09+0800 - 63.93 MB
                  _1ri.fnm: 2014-11-21T12:29:10+0800 - 18.00 B
                  _1ri.frq: 2014-11-21T12:29:21+0800 - 61.92 MB
                  _1ri.nrm: 2014-11-21T12:29:22+0800 - 7.99 MB
                  _1ri.prx: 2014-11-21T12:29:25+0800 - 15.98 MB
                  _1ri.tii: 2014-11-21T12:29:25+0800 - 1.84 MB
                  _1ri.tis: 2014-11-21T12:29:48+0800 - 129.59 MB
                  _2lv.fdt: 2014-11-21T12:30:33+0800 - 303.36 MB
                  _2lv.fdx: 2014-11-21T12:30:01+0800 - 73.55 MB
                  _2lv.fnm: 2014-11-21T12:30:01+0800 - 18.00 B
                  _2lv.frq: 2014-11-21T12:30:14+0800 - 71.53 MB
                  _2lv.nrm: 2014-11-21T12:30:16+0800 - 9.19 MB
                  _2lv.prx: 2014-11-21T12:30:19+0800 - 18.39 MB
                  _2lv.tii: 2014-11-21T12:30:20+0800 - 2.14 MB
                  _2lv.tis: 2014-11-21T12:30:46+0800 - 150.79 MB
                  _3gr.fdt: 2014-11-21T12:31:28+0800 - 311.85 MB
                  _3gr.fdx: 2014-11-21T12:31:00+0800 - 75.61 MB
                  _3gr.fnm: 2014-11-21T12:31:00+0800 - 18.00 B
                  _3gr.frq: 2014-11-21T12:31:13+0800 - 73.59 MB
                  _3gr.nrm: 2014-11-21T12:31:15+0800 - 9.45 MB
                  _3gr.prx: 2014-11-21T12:31:18+0800 - 18.90 MB
                  _3gr.tii: 2014-11-21T12:31:19+0800 - 2.18 MB
                  _3gr.tis: 2014-11-21T12:31:46+0800 - 154.25 MB
                  _4ah.fdt: 2014-11-21T12:32:23+0800 - 306.20 MB
                  _4ah.fdx: 2014-11-21T12:31:59+0800 - 74.23 MB
                  _4ah.fnm: 2014-11-21T12:31:59+0800 - 18.00 B
                  _4ah.frq: 2014-11-21T12:32:12+0800 - 72.22 MB
                  _4ah.nrm: 2014-11-21T12:32:14+0800 - 9.28 MB
                  _4ah.prx: 2014-11-21T12:32:17+0800 - 18.56 MB
                  _4ah.tii: 2014-11-21T12:32:17+0800 - 2.14 MB
                  _4ah.tis: 2014-11-21T12:32:44+0800 - 152.60 MB
                  _4d9.cfs: 2014-11-21T12:32:34+0800 - 65.42 MB
                  _4g1.cfs: 2014-11-21T12:32:46+0800 - 65.54 MB
                  _4hz.cfs: 2014-11-21T12:32:46+0800 - 6.42 MB
                  _4ij.cfs: 2014-11-21T12:32:47+0800 - 6.44 MB
                  _4it.cfs: 2014-11-21T12:32:57+0800 - 65.35 MB
                  _4ll.cfs: 2014-11-21T12:32:58+0800 - 65.73 MB
                  _4od.cfs: 2014-11-21T12:33:09+0800 - 65.89 MB
                  _4on.cfs: 2014-11-21T12:33:00+0800 - 6.50 MB
                  _4ox.cfs: 2014-11-21T12:33:01+0800 - 6.57 MB
                  _4p7.cfs: 2014-11-21T12:33:02+0800 - 6.57 MB
                  _4ph.cfs: 2014-11-21T12:33:03+0800 - 6.60 MB
                  _4pr.cfs: 2014-11-21T12:33:05+0800 - 6.60 MB
                  _4px.cfs: 2014-11-21T12:33:05+0800 - 718.54 kB
                  _4py.cfs: 2014-11-21T12:33:05+0800 - 723.03 kB
                  _4pz.cfs: 2014-11-21T12:33:05+0800 - 719.12 kB
                  _4q1.cfs: 2014-11-21T12:33:07+0800 - 6.59 MB
                  _gf.fdt: 2014-11-21T12:33:29+0800 - 131.96 MB
                  _gf.fdx: 2014-11-21T12:33:15+0800 - 33.04 MB
                  _gf.fnm: 2014-11-21T12:33:15+0800 - 18.00 B
                  _gf.frq: 2014-11-21T12:33:20+0800 - 31.02 MB
                  _gf.nrm: 2014-11-21T12:33:21+0800 - 4.13 MB
                  _gf.prx: 2014-11-21T12:33:23+0800 - 8.26 MB
                  _gf.tii: 2014-11-21T12:33:23+0800 - 1.01 MB
                  _gf.tis: 2014-11-21T12:33:33+0800 - 73.32 MB
                - Total: 2014-11-24T10:06:47+0800 - 3.51 GB
              - Total: 2014-11-21T12:28:14+0800 - 3.51 GB
            - Total: 2014-11-21T12:28:14+0800 - 3.51 GB
            lucene-store.db: 2014-11-22T17:39:30+0800 - 40.00 B
            lucene.log.1: 2014-11-21T18:34:32+0800 - 16.00 B
            lucene.log.active: 2014-11-21T18:33:31+0800 - 4.00 B
          - Total: 2014-11-22T11:40:56+0800 - 3.51 GB
          index.db: 2014-11-21T08:52:48+0800 - 103.00 B
          lock: 2014-11-22T15:32:04+0800 - 0.00 B
          messages.log: 2014-11-24T10:06:44+0800 - 37.60 MB
          neostore: 2014-11-24T10:06:42+0800 - 54.00 B
          neostore.id: 2014-11-24T10:06:47+0800 - 9.00 B
          neostore.nodestore.db: 2014-11-22T15:32:04+0800 - 473.40 MB
          neostore.nodestore.db.id: 2014-11-24T10:06:47+0800 - 9.00 B
          neostore.propertystore.db: 2014-11-22T15:32:04+0800 - 2.30 GB
          neostore.propertystore.db.arrays: 2014-11-22T15:32:04+0800 - 128.00 B
          neostore.propertystore.db.arrays.id: 2014-11-24T10:06:46+0800 - 9.00 B
          neostore.propertystore.db.id: 2014-11-24T10:06:46+0800 - 9.00 B
          neostore.propertystore.db.index: 2014-11-22T17:03:23+0800 - 5.22 MB
          neostore.propertystore.db.index.id: 2014-11-24T10:06:46+0800 - 9.00 B
          neostore.propertystore.db.index.keys: 2014-11-22T15:32:04+0800 - 1024.00 kB
          neostore.propertystore.db.index.keys.id: 2014-11-24T10:06:42+0800 - 9.00 B
          neostore.propertystore.db.strings: 2014-11-22T15:32:04+0800 - 204.75 kB
          neostore.propertystore.db.strings.id: 2014-11-24T10:06:46+0800 - 9.00 B
          neostore.relationshipstore.db: 2014-11-22T15:32:04+0800 - 20.66 GB
          neostore.relationshipstore.db.id: 2014-11-24T10:06:46+0800 - 9.00 B
          neostore.relationshiptypestore.db: 2014-11-24T10:06:42+0800 - 10.00 B
          neostore.relationshiptypestore.db.id: 2014-11-24T10:06:42+0800 - 9.00 B
          neostore.relationshiptypestore.db.names: 2014-11-22T15:32:04+0800 - 114.00 B
          neostore.relationshiptypestore.db.names.id: 2014-11-24T10:06:42+0800 - 9.00 B
          nioneo_logical.log.1: 2014-11-22T15:32:23+0800 - 16.00 B
          nioneo_logical.log.active: 2014-11-22T15:32:04+0800 - 4.00 B
          nioneo_logical.log.v4729: 2014-11-22T15:31:23+0800 - 16.00 B
          store_lock: 2014-11-21T12:21:37+0800 - 0.00 B
          tm_tx_log.1: 2014-11-24T10:06:47+0800 - 0.00 B
          tm_tx_log.2: 2014-11-21T12:21:38+0800 - 28.68 kB
    2014-11-24 02:06:47.630+0000 INFO  [org.neo4j]: --- STARTED diagnostics for KernelDiagnostics:StoreFiles END ---
    2014-11-24 02:06:47.630+0000 INFO  [org.neo4j]: Started - database is now available
    2014-11-24 02:06:47.630+0000 INFO  [org.neo4j]: GC Monitor started. 
    2014-11-24 02:24:00.340+0000 INFO  [org.neo4j]: GC Monitor: Application threads blocked for an additional 142ms [total block time: 0.142s]
    2014-11-24 02:42:44.771+0000 INFO  [org.neo4j]: GC Monitor: Application threads blocked for an additional 373ms [total block time: 0.515s]

neo4j.properties:

dump_configuration=false
use_memory_mapped_buffers:true
neostore.nodestore.db.mapped_memory=100M
neostore.relationshipstore.db.mapped_memory=100M
neostore.propertystore.db.mapped_memory=100M
neostore.propertystore.db.strings.mapped_memory=20M
neostore.propertystore.db.arrays.mapped_memory=20M   /* file cache*/
neostore.propertystore.db.index.keys.mapped_memory=20M
neostore.propertystore.db.index.mapped_memory=20M
cache_type=soft
cache.memory_radio=50%
node_cache_size=2000M
relationship_cache_size=800M 
excution_guard_enabled=true
keep_logical_logs=false
online_backup_enabled=true
ha.pull_interval=10

1 个答案:

答案 0 :(得分:0)

Direction.BOTH表示它是圆圈。

您的MMIO设置应该更大(更接近文件大小), 你的堆也可能(因为它包括Windows上的mmio部分 - &gt;至少12G到16G堆)。

您没有共享messages.log中有关堆大小的指示。

neostore.nodestore.db.mapped_memory=500M
neostore.relationshipstore.db.mapped_memory=6G
neostore.propertystore.db.mapped_memory=1G
neostore.propertystore.db.strings.mapped_memory=1G

如果您仍然运行企业,则应使用:

cache_type=hpc 

cache_type=none

对于Guard,除了设置你已经拥有的配置外,添加它的代码是这样的:

((GraphDatabaseAPI)graphDB).getDependencyResolver().resolveDependency(Guard.class);

guard.startTimeout(timeLimit);
guard.startOperationsCount(maxOps);
try {
    return doOperation();
} catch (GuardException e) {
    LOG.warn("Aborting Operation ");
    throw new RuntimeException(e); 
} finally {
    guard.stop();
}