Question

我刚刚开始评估Neo4j，看它是否适合我们的用例。我正在使用嵌入式Java API将边和节点插入图形中。创建大约5000个节点后，我得到以下错误（在OS X Yosemite上使用Neo4j 2.1.6和2.1.7）

org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction

Caused by: javax.transaction.xa.XAException

Caused by: org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: java.io.FileNotFoundException: /Users/mihir.k/IdeaProjects/Turant/target/neo4j-hello-db/schema/label/lucene/_8zr.frq (Too many open files)

Caused by: java.io.FileNotFoundException: /Users/mihir.k/IdeaProjects/Turant/target/neo4j-hello-db/schema/label/lucene/_8zr.frq (Too many open files)

我在线查看了许多类似的StackOverFlow问题和其他相关主题。他们都建议增加最大打开文件限制。我试过这样做。

这些是我的设置：

kern.maxfiles：65536
kern.maxfilesperproc：65536

然而，这并未修复错误。

当Neo4j代码运行时，我尝试使用lsof|wc -l命令。当大约10000个文件打开时，代码总是会中断。

以下是处理Neo4j的主要类：

import java.io.File;
import java.io.Serializable;
import java.util.Map;
import java.util.concurrent.TimeUnit;

import org.neo4j.cypher.internal.compiler.v1_9.commands.True;
import org.neo4j.cypher.internal.compiler.v2_0.ast.False;
import org.neo4j.graphdb.*;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;
import org.neo4j.graphdb.schema.Schema;
import org.neo4j.graphdb.schema.IndexDefinition;
import org.neo4j.graphdb.index.UniqueFactory;
import org.neo4j.graphdb.index.Index;
import org.neo4j.graphdb.index.IndexHits;

public class Neo4jDB implements  Serializable {

    private static final String DB_PATH = "target/neo4j-hello-db-spark";
    IndexDefinition indexDefinition;

    private static GraphDatabaseFactory dbFactory;
    public static GraphDatabaseService db;

    public void main(String[] args) {
        System.out.println("Life is a disease, sexually transmitted and irrevocably fatal. Stop coding and read some Neil Gaiman.");
    }



    public void startDbInstance() {
        db =new GraphDatabaseFactory().newEmbeddedDatabase(DB_PATH);
    }

    public  Node createOrGetNode ( LabelsUser360 label , String key, String nodeName ,Map<String,Object> propertyMap)
    {
        System.out.println("Creating/Getting node");
        try ( Transaction tx = db.beginTx() ) {
            Node node;

            if (db.findNodesByLabelAndProperty(label, key, nodeName).iterator().hasNext()) {
                node = db.findNodesByLabelAndProperty(label, key, nodeName).iterator().next();
            } else {
                node = db.createNode(label);
                node.setProperty(key, nodeName);

            }
            for (Map.Entry<String, Object> entry : propertyMap.entrySet()) {
                node.setProperty(entry.getKey(), entry.getValue());
            }
            tx.success();
            return node;
        }

    }

    public  void createUniquenessConstraint(LabelsUser360 label , String property)
    {
        try ( Transaction tx = db.beginTx() )
        {
            db.schema()
                    .constraintFor(label)
                    .assertPropertyIsUnique(property)
                    .create();
            tx.success();
        }
    }

    public void createOrUpdateRelationship(RelationshipsUser360 relationshipType ,Node startNode, Node endNode, Map<String,Object> propertyMap)
    {
        try ( Transaction tx = db.beginTx() ) {
            if (startNode.hasRelationship(relationshipType, Direction.OUTGOING)) {
                Relationship relationship = startNode.getSingleRelationship(relationshipType, Direction.OUTGOING);
                for (Map.Entry<String, Object> entry : propertyMap.entrySet()) {
                    relationship.setProperty(entry.getKey(), entry.getValue());
                }
            } else {
                Relationship relationship = startNode.createRelationshipTo(endNode, relationshipType);
                for (Map.Entry<String, Object> entry : propertyMap.entrySet()) {
                    relationship.setProperty(entry.getKey(), entry.getValue());
                }
            }
            tx.success();
        }

    }

    public void registerShutdownHook( final GraphDatabaseService graphDb )
    {
        Runtime.getRuntime().addShutdownHook( new Thread()
        {
            @Override
            public void run()
            {
                db.shutdown();
            }
        } );
    }

}

还有另一个Neo4jAdapter类用于实现特定于域的逻辑。它使用Neo4jDB类来添加/更新节点/属性/关系

import org.apache.lucene.index.IndexWriter;
import org.codehaus.jackson.map.ObjectMapper;
import org.json.*;
import org.neo4j.graphdb.*;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;
import org.neo4j.graphdb.schema.IndexDefinition;

import java.io.*;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;


public class Neo4jAdapter implements Serializable {

    static Neo4jDB n4j = new Neo4jDB();
    public static GraphDatabaseService db = Neo4jDB.db ;

    public void begin() {
        n4j.startDbInstance();
    }


    public static void main(String[] args) {}



    public String graphPut(String jsonString) {

        System.out.println("graphput called");

        HashMap<String, Object> map = jsonToMap(jsonString); //Json deserializer

        Node startNode = n4j.createOrGetNode(...);
        Node endNode = n4j.createOrGetNode(...);

        propertyMap = new HashMap<String, Object>();
        propertyMap.put(....);

        try (Transaction tx = Neo4jDB.db.beginTx()) {
            Relationship relationship = startNode.getSingleRelationship(...);
            if (relationship != null) {
                Integer currentCount = (Integer) relationship.getProperty("count");
                Integer updatedCount = currentCount + 1;
                propertyMap.put("count", updatedCount);
            } else {
                Integer updatedCount = 1;
                propertyMap.put("count", updatedCount);
            }
            tx.success();
        }


        n4j.createOrUpdateRelationship(RelationshipsUser360.BLAH, startNode, endNode, propertyMap);

                }
            }
        }
        return "Are you sponge worthy??";
    }
}

最后，有一个Sprak应用程序调用Neo4jAdapter类的“graphput”方法。相关的代码片段（以下是scala + spark代码）：

val graphdb : Neo4jAdapter = new Neo4jAdapter()
graphdb.begin() 
linesEnriched.foreach(a=>graphdb.graphPutMap(a))

其中'a'是json字符串，linesEnriched是Spark RDD（基本上是一组字符串）

由Lucene引起的Neo4j错误（太多打开的文件）

0 个答案: