Neo4j:并发和内存映射缓冲区

时间:2014-05-23 13:44:38

标签: java multithreading neo4j

我正在嵌入式Neo4J DB 2.0.3上启动一批遍历。

  • 23M节点
  • 87M关系
  • 16GB堆
  • 尝试了不同的缓存设置
  • 尝试了不同的线程设置(从20到5个线程)

Job正常运行一段时间(例如:1h)然后吞吐量急剧减慢,因为线程花费大部分时间等待锁定。

看起来内存缓冲区(MappedPersistenceWindow)不能被多个线程共享,这听起来有点奇怪。

来自线程转储的一些示例:

"taskExecutor-6" - Thread t@17
   java.lang.Thread.State: BLOCKED
  at java.lang.Object.wait(Native Method)
  - waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
  at java.lang.Object.wait(Object.java:503)
  at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
  at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
  at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
  at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
  at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
  at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
  at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
  at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
  at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
  at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
  at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
  at org.neo4j.kernel.impl.core.RelationshipProxy.getType(RelationshipProxy.java:141)


"taskExecutor-5" - Thread t@16
   java.lang.Thread.State: RUNNABLE
  at org.neo4j.kernel.impl.nioneo.store.LockableWindow.markAsInUse(LockableWindow.java:70)
  - locked <6abe6e1> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
  at org.neo4j.kernel.impl.nioneo.store.BrickElement.getAndMarkWindow(BrickElement.java:94)
  at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:147)
  at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
  at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
  at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
  at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
  at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
  at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
  at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
  - locked <2b29ed5b> (a org.neo4j.kernel.impl.core.NodeImpl)
  at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
  at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
  at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
  at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
  at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)


"taskExecutor-4" - Thread t@15
   java.lang.Thread.State: BLOCKED
  at java.lang.Object.wait(Native Method)
  - waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
  at java.lang.Object.wait(Object.java:503)
  at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
  at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
  at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
  at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
  at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
  at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
  at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
  at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
  at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
  - locked <41fe8c4f> (a org.neo4j.kernel.impl.core.NodeImpl)
  at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
  at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
  at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
  at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
  at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)


"taskExecutor-3" - Thread t@14
   java.lang.Thread.State: WAITING
  at java.lang.Object.wait(Native Method)
  - waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
  at java.lang.Object.wait(Object.java:503)
  at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
  at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
  at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
  at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
  at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
  at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
  at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
  at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
  at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
  at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
  at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
  at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
  at org.neo4j.kernel.impl.core.RelationshipProxy.getOtherNode(RelationshipProxy.java:108)
  at org.neo4j.kernel.impl.traversal.TraversalBranchImpl.next(TraversalBranchImpl.java:145)
  at org.neo4j.graphdb.traversal.PreorderDepthFirstSelector.next(PreorderDepthFirstSelector.java:49)
  at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:68)
  at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:35)
  at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)

有什么想法吗?

0 个答案:

没有答案