在OrientDB中插入大量边缘会导致崩溃(ODatabaseException)

时间:2015-06-23 15:19:32

标签: database insert bulkinsert orientdb

我在OrientDB中插入大量(几百万)边缘时遇到问题。在java中,我有一个包含~5000行和列的矩阵,其中每个条目表示对象之间的相似性值。我想将此信息插入OrientDB。

但是在创建了200k到400k之间的边缘之后,发生了ODatabaseException,我还没有找到解决这个问题的方法。

我正在通过以下方式创建边缘:使用SQL Batch,我将SQL查询添加到字符串并为每100个边执行批处理。每个SQL行都如下所示:

CEATE EDGE similarity FROM #10:0 TO #10:1 SET score = 0.5

我应该以不同的方式做这件事吗?是缺乏记忆问题吗?我一直无法找到解决方案。

感谢。

修改:

我正在使用异常和一些代码更新问题:

代码:

public void saveHouseCharacteristicsSimilarities(double[][] houseSim,
        ArrayList<OHouse> houseList) {

    OrientGraphFactory factory = new OrientGraphFactory("plocal:/directory");
    OrientGraphNoTx graph = factory.getNoTx();

    int counter=0;
    String query = "BEGIN\n";
    for (int h1 = 0; h1 < (houseList.size() - 1); h1++) {
        for (int h2 = (h1 + 1); h2 < houseList.size(); h2++) {
            counter++;

            query += "create edge similarity from "
                    + houseList.get(h1).getRid() + " to "
                    + houseList.get(h2).getRid() + " SET score = "
                    + houseSim[h1][h2] + "\n";

            if (counter % 100 == 0
                    || ((h1 == (houseList.size() - 1)) && (h2 == (houseList
                            .size() - 1)))) {
                query += "COMMIT";
                graph.command(new OCommandScript("sql", query)).execute();
                query = "BEGIN";
            }
        }
    }
    graph.shutdown();
}

异常(在此特定点我已经保存了250000条边):

  com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #24:1880 (cluster: house)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1605)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:226)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1424)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:117)
    at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:313)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.getVertex(OrientBaseGraph.java:731)
    at com.orientechnologies.orient.graph.sql.OCommandExecutorSQLCreateEdge$1.call(OCommandExecutorSQLCreateEdge.java:144)
    at com.orientechnologies.orient.graph.sql.OCommandExecutorSQLCreateEdge$1.call(OCommandExecutorSQLCreateEdge.java:126)
    at com.orientechnologies.orient.graph.sql.OGraphCommandExecutorSQLFactory.runInTx(OGraphCommandExecutorSQLFactory.java:123)
    at com.orientechnologies.orient.graph.sql.OGraphCommandExecutorSQLFactory.runInTx(OGraphCommandExecutorSQLFactory.java:141)
    at com.orientechnologies.orient.graph.sql.OCommandExecutorSQLCreateEdge.execute(OCommandExecutorSQLCreateEdge.java:126)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:64)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:1184)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:1173)
    at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:63)
    at com.orientechnologies.orient.core.command.script.OCommandExecutorScript.executeSQLScript(OCommandExecutorScript.java:335)
    at com.orientechnologies.orient.core.command.script.OCommandExecutorScript.executeSQL(OCommandExecutorScript.java:138)
    at com.orientechnologies.orient.core.command.script.OCommandExecutorScript.executeInContext(OCommandExecutorScript.java:82)
    at com.orientechnologies.orient.core.command.script.OCommandExecutorScript.execute(OCommandExecutorScript.java:73)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:1184)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:1173)
    at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:63)
    at com.tinkerpop.blueprints.impls.orient.OrientGraphCommand.execute(OrientGraphCommand.java:48)
    at com.ubiwhere.livinx.recEngine.dao.OHouseDAO.saveHouseCharacteristicsSimilarities(OHouseDAO.java:465)
    at com.ubiwhere.livinx.recEngine.structures.CharacteristicsSimilarity.buildCharacteristicsMatrix(CharacteristicsSimilarity.java:33)
    at com.ubiwhere.livinx.rest.house.HouseFavouriteService.getFavourites(HouseFavouriteService.java:195)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:415)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
    at org.glassfish.grizzly.ProcessorExecutor.execute(Proc

2 个答案:

答案 0 :(得分:0)

尝试这样的东西来创建边

OrientVertex vertex = graph.getVertex(houseList.get(h1).getRid());
OrientVertex vertex2 = graph.getVertex(houseList.get(h2).getRid());
vertex.addEdge("similarity",vertex2,new Object[]{"score",houseSim[h1][h2]});

答案 1 :(得分:0)

好吧,问题似乎不是东方本身,而是程序运行的glassfish服务器。由于某种原因,Glassfish终止了与东方的联系。无论如何,在服务器外运行程序解决了这个问题。谢谢!