Neo4j Unmanaged Extension和GuardTimeoutException

时间：2014-10-05 00:01:16

标签： java neo4j jersey cypher graph-databases

我非常需要一些关于我正在使用Java构建的Neo4j非托管扩展的问题的建议。我创建了一个非常简单的代码示例，突出了我的问题。基本前提是我想设置

org.neo4j.server.webserver.limit.executiontime

neo4j服务器在合理的时间内进行用户查询（比方说2分钟），这些查询通过Cypher，其他端点等进入。但是我还有一些批处理作业，我需要通过这个非托管扩展来运行，因此我试图将它们分成几个＆lt; 2分钟的交易。我看到的问题是，即使我的每个事务都是<2分钟，但是一旦我的进程运行了2分钟，我就会得到一个GuardTimeoutException。

以下是样本。在这个示例中，我将时间限制在2000毫秒，因此我不需要整天调试。 （虽然变得非常接近！）

端点

/**
 * Sample endpoint.
@GET
@Path("test")
@Produces(MediaType.APPLICATION_JSON)
public Response test(@Context GraphDatabaseService service) {
    TestFile.test(service);
    return Response.ok().build();
}

逻辑

public static void test(final GraphDatabaseService service) {
    for (int i = 0; i < 100000; i++) {
        try (Transaction tx = service.beginTx();) {
            final Node n = service.createNode();
            n.addLabel(testLabel);
            tx.success();
            tx.close();
        }
        System.out.println("Added node");
    }
}

我可以看到每个事务只需要几分之一秒，因为我在超时之前成功加载了大约200个节点。然而，在击中终点时仅2秒，我得到以下结果：

org.neo4j.kernel.guard.GuardTimeoutException: timeout occured (overtime=1)
at org.neo4j.kernel.guard.Guard$Timeout.check(Guard.java:132)
at org.neo4j.kernel.guard.Guard.check(Guard.java:43)
at org.neo4j.kernel.InternalAbstractGraphDatabase$5.createNode(InternalAbstractGraphDatabase.java:794)
at org.neo4j.kernel.impl.api.state.OldTxStateBridgeImpl.nodeCreate(OldTxStateBridgeImpl.java:120)
at org.neo4j.kernel.impl.api.state.TxStateImpl.nodeDoCreate(TxStateImpl.java:366)
at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.nodeCreate(StateHandlingStatementOperations.java:99)
at org.neo4j.kernel.impl.api.ConstraintEnforcingEntityOperations.nodeCreate(ConstraintEnforcingEntityOperations.java:390)
at org.neo4j.kernel.impl.api.LockingStatementOperations.nodeCreate(LockingStatementOperations.java:207)
at org.neo4j.kernel.impl.api.OperationsFacade.nodeCreate(OperationsFacade.java:506)
at org.neo4j.kernel.InternalAbstractGraphDatabase.createNode(InternalAbstractGraphDatabase.java:1120)
at **TestFile.test(TestFile.java:15)
at **test(Jobs.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at org.neo4j.server.rest.transactional.TransactionalRequestDispatcher.dispatch(TransactionalRequestDispatcher.java:139)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1506)
at org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
at org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:211)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:445)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:744)

正如你所看到的，我已经尝试了tx.success（），tx.close（），我能想到的任何事情都可以让所有事务在完成时死掉。任何建议都非常感谢!!

更新-------------

迈克尔 - 我已经按照你建议的步骤进行了操作 - 我有一个新的java类，它扩展了SPIPluginLifecycle，并在/ src / main / resources / META-INF / services中添加了新的配置文件。我可以看到这个文件最终出现在META-INF / services中的/ plugins中的jar中。但是，我根本没有看到这个初始化类被调用。下面是我在启动时看到的类和日志。 

public class GraphInitializer implements SPIPluginLifecycle {
private WebServer webServer;

@Override
public Collection<Injectable<?>> start(final GraphDatabaseService graphDatabaseService, final Configuration config) {
    throw new IllegalAccessError();
}

@Override
public void stop() {

}

@Override
public Collection<Injectable<?>> start(final NeoServer neoServer) {
    System.out.println("K starting!");
    webServer = getWebServer(neoServer);
    final Database database = neoServer.getDatabase();
    final GraphDatabaseAPI graphDatabaseAPI = database.getGraph();
    final Guard guard = graphDatabaseAPI.getDependencyResolver().resolveDependency(Guard.class);
    final Filter filter = new GuardingRequestFilter(guard, 3000);
    webServer.addFilter(filter, "/*" );
    return null;
}

private WebServer getWebServer(final NeoServer neoServer) {
    if (neoServer instanceof AbstractNeoServer) {
        return ((AbstractNeoServer) neoServer).getWebServer();
    }
    throw new IllegalArgumentException("expected AbstractNeoServer");
}

日志：

2014-10-06 16:14:23.009+0000 INFO  [API] Mounted discovery module at [/]
2014-10-06 16:14:23.014+0000 INFO  [API] Mounted REST API at [/db/data/]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted management API at [/db/manage/]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted third-party JAX-RS package [***] at [/kristen]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted webadmin at [/webadmin]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted Neo4j Browser at [/browser]
2014-10-06 16:14:23.070+0000 INFO  [API] Mounting static content at [/webadmin] from [webadmin-html]
2014-10-06 16:14:23.124+0000 INFO  [API] Mounting static content at [/browser] from [browser]
12:14:23.126 [main] WARN  o.e.j.server.handler.ContextHandler - o.e.j.s.ServletContextHandler@1cb1c025{/,null,null} contextPath ends with /
12:14:23.127 [main] WARN  o.e.j.server.handler.ContextHandler - Empty contextPath
12:14:23.131 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.0.5.v20130815
12:14:23.155 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.h.MovedContextHandler@5de15910{/,null,AVAILABLE}
12:14:23.245 [main] INFO  o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /webadmin, did not find org.apache.jasper.servlet.JspServlet
12:14:23.255 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@4b45c74e{/webadmin,jar:file:/usr/local/Cellar/neo4j/2.1.3/libexec/system/lib/neo4j-server-2.1.3-static-web.jar!/webadmin-html,AVAILABLE}
12:14:23.668 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@2917eb65{/kristen,null,AVAILABLE}
12:14:23.817 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@54589209{/db/manage,null,AVAILABLE}
12:14:24.003 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@68a1f8e7{/db/data,null,AVAILABLE}
12:14:24.021 [main] INFO  o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /browser, did not find org.apache.jasper.servlet.JspServlet
12:14:24.022 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@438e0daa{/browser,jar:file:/usr/local/Cellar/neo4j/2.1.3/libexec/system/lib/neo4j-browser-2.1.3.jar!/browser,AVAILABLE}
12:14:24.103 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@1cb1c025{/,null,AVAILABLE}
12:14:24.115 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@75b61dd7{HTTP/1.1}{localhost:7474}
12:14:24.503 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@6464d61f{SSL-HTTP/1.1}{localhost:7473}
2014-10-06 16:14:24.503+0000 INFO  [API] Server started on: http://localhost:7474/
2014-10-06 16:14:24.504+0000 INFO  [API] Remote interface ready and available at [http://localhost:7474/]

我期待一个新的行条目或其他内容 - 我的更改添加暂停实际上不起作用，所以我确定这些更改尚未生效。我还有什么需要添加到neo4j-server.properties，neo4j.properties等吗？我已经成功添加了设置非托管扩展的行。

谢谢你！

2 个答案:

答案 0 :(得分：1)

小事务使它变得更慢，因为每个事务都必须刷新到磁盘。 通常，在一个tx中添加多达50k个节点是有意义的，这应该不到一秒钟。

对于你的保护过滤器，它只是一个servlet过滤器，所以你也可以检查安装它的Neo4j源代码，并在你的扩展的init上自己安装，但是要排除要批处理的端点。

在：org.neo4j.server.AbstractNeoServer第488行

    Filter filter = new GuardingRequestFilter( guard,
            getConfiguration().getInt( Configurator.WEBSERVER_LIMIT_EXECUTION_TIME_PROPERTY_KEY ) );
    webServer.addFilter( filter, "/*" );

您可以使用扩展程序初始化程序添加自定义过滤器，如下所示：

https://github.com/neo4j-contrib/authentication-extension/blob/2.1/src/main/java/org/neo4j/server/extension/auth/AuthenticationExtensionInitializer.java#L81

你必须添加一个包含初始化程序类名的文件：

https://github.com/neo4j-contrib/authentication-extension/blob/2.1/src/main/resources/META-INF/services/org.neo4j.server.plugins.PluginLifecycle

答案 1 :(得分：0)

解决方案Michael Hunger发布的工作非常完美，并且还有一些我自己的学习要点。

  1. 我必须将neo4j-server.properties中的org.neo4j.server.thirdparty_jaxrs_classes属性更新为更多的根目录，因为我没有将我的初始化程序放在与我的api java文件相同的目录中。
  2. start（）方法必须返回一个空集合。
  3. 我能够使用多个不同的过滤器来指定不同端点的时间限制。

    4. 下面的最终代码：

    @Override
public Collection<Injectable<?>> start(final NeoServer neoServer) {
    webServer = getWebServer(neoServer);
    final Database database = neoServer.getDatabase();
    final GraphDatabaseAPI graphDatabaseAPI = database.getGraph();
    final Guard guard = graphDatabaseAPI.getDependencyResolver().resolveDependency(Guard.class);
    final Filter filter = new GuardingRequestFilter(guard, 600);
    webServer.addFilter(filter, "/*" );
    final Filter filter2 = new GuardingRequestFilter(guard, 3000);
    webServer.addFilter(filter2, "*/batch" );
    return Collections.emptyList();
}
相关问题
最新问题