Java Neo4J内存不足

时间:2012-06-19 14:43:55

标签: java memory neo4j

这有点像这样:Neo4j OutOfMemory problem

但它已经过时了,据我所知,解决方案显然也是如此。

所以我试图插入大约100K节点与5.5M关系(我实际上减少了我的数据集,所以它现在更像是< 100K节点,关系为2.8M)。

过了一会儿,内存耗尽,我得到一个例外:

Exception in thread "GC-Monitor" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at java.lang.StringBuilder.toString(Unknown Source)
    at org.neo4j.kernel.impl.util.StringLogger$ActualStringLogger.logMessage(StringLogger.java:276)
    at org.neo4j.kernel.impl.cache.MeasureDoNothing.run(MeasureDoNothing.java:85)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.LinkedList.addBefore(Unknown Source)
    at java.util.LinkedList.add(Unknown Source)
    at org.neo4j.kernel.impl.nioneo.store.IdGeneratorImpl.freeId(IdGeneratorImpl.java:291)
    at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.freeId(CommonAbstractStore.java:382)
    at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doRollback(WriteTransaction.java:315)
    at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.rollback(XaTransaction.java:278)
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:518)
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111)
    at org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:558)
    at org.neo4j.kernel.impl.transaction.TxManager.rollback(TxManager.java:610)
    at org.neo4j.kernel.impl.transaction.TransactionImpl.rollback(TransactionImpl.java:129)
    at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:119)
    at sqlToGraph.SqlToGraph.main(SqlToGraph.java:81)

我已经尝试将-Xmx1500m传递给java,这是我可以传递的限制,因为在它抱怨无法分配堆空间之前。它会持续很长时间,但仍然无法完成。

以下是(稍加编辑的)代码:

/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
    while (rs.next()) {
        user_lo = rs.getInt(1);
        user_hi = rs.getInt(2);
        n_lo = getOrCreate(user_lo, graphDb);
        n_lo.setProperty("user_id", user_lo);
        n_lo.setProperty("total", rs.getInt(3));
        n_hi = getOrCreate(user_hi, graphDb);
        n_hi.setProperty("user_id", user_hi);
        n_hi.setProperty("total", rs.getInt(4));
        relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
        relationship.setProperty("mean_percent", rs.getDouble(5));
    }
    tx.success();
} finally {
    tx.finish();
}
graphDb.shutdown();

2 个答案:

答案 0 :(得分:11)

在这里添加另一个答案。因此,给定代码的问题是您永远不会提交您的事务。事务数据保存在内存中直到提交,因此所有创建的节点和关系将只放在内存中等待提交,这就是您最终获得OOM的原因。

我建议改变代码:

/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
    for (int i = 0; rs.next(); i++) {
        user_lo = rs.getInt(1);
        user_hi = rs.getInt(2);
        n_lo = getOrCreate(user_lo, graphDb);
        n_lo.setProperty("user_id", user_lo);
        n_lo.setProperty("total", rs.getInt(3));
        n_hi = getOrCreate(user_hi, graphDb);
        n_hi.setProperty("user_id", user_hi);
        n_hi.setProperty("total", rs.getInt(4));
        relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
        relationship.setProperty("mean_percent", rs.getDouble(5));

        // Commit every now and then to free memory.
        if ( i > 0 && i % 10000 == 0 ) {
            tx.success();
            tx.finish();
            tx = graphDb.beginTx();
        }
    }
    tx.success();
} finally {
    tx.finish();
}
graphDb.shutdown();

答案 1 :(得分:0)

如果依赖于单个事务的原子性保证,拆分事务对您来说是不可行的,另一个选择可能是简化数据模型,例如:减少添加到节点/关系的属性数量。

在我们的案例中,我们可以通过省略保存默认值或零的属性来解决OOM问题。