Question

我有一个用Java语言开发的简单Web应用程序，它利用DL4J和ND4J的库依赖来读取和处理某个文本文件（文本文件的大小很大，大约100- 200mb）位于WAR文件的WEB-INF文件夹内。我正在使用Maven构建工具，并且在POM.xml中包含了所有必需的依赖项：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>com.example</groupId>
<artifactId>simple-heroku-webapp</artifactId>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>
<name>simple-heroku-webapp</name>


<properties>
    <nd4j.version>0.6.0</nd4j.version>
    <dl4j.version>0.6.0</dl4j.version>
    <jersey.version>2.23.2</jersey.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <jetty.version>9.0.6.v20130930</jetty.version>
</properties>


<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.glassfish.jersey</groupId>
            <artifactId>jersey-bom</artifactId>
            <version>${jersey.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>



<dependencies>
    <dependency>
        <groupId>org.glassfish.jersey.containers</groupId>
        <artifactId>jersey-container-servlet</artifactId>
    </dependency>

    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-servlet</artifactId>
        <version>${jetty.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-webapp</artifactId>
        <version>${jetty.version}</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.glassfish.jersey.test-framework.providers</groupId>
        <artifactId>jersey-test-framework-provider-bundle</artifactId>
        <type>pom</type>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-ui</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-nlp</artifactId>
        <version>${dl4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>nd4j-native</artifactId>
        <version>${nd4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>nd4j-common</artifactId>
        <version>${nd4j.version}</version>
    </dependency>
</dependencies>

<build>
    <finalName>simple-heroku-webapp</finalName>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.5.1</version>
            <inherited>true</inherited>
            <configuration>
                <source>1.7</source>
                <target>1.7</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>copy-dependencies</id>
                    <phase>package</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <includeScope>compile</includeScope>
                    </configuration>
                </execution>
                <execution>
                    <phase>package</phase>
                    <goals><goal>copy</goal></goals>
                    <configuration>
                        <artifactItems>
                            <artifactItem>
                                <groupId>org.eclipse.jetty</groupId>
                                <artifactId>jetty-runner</artifactId>
                                <version>9.0.0.M5</version>
                                <destFileName>jetty-runner.jar</destFileName>
                            </artifactItem>
                        </artifactItems>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-maven-plugin</artifactId>
            <version>${jetty.version}</version>
            <configuration>
                <contextPath>/</contextPath>
                <webApp>
                    <contextPath>/</contextPath>
                    <webInfIncludeJarPattern>.*/.*jersey-[^/]\.jar$</webInfIncludeJarPattern>
                </webApp>
                <war>${project.build.directory}/${project.build.finalName}.war</war>
            </configuration>
        </plugin>
        <plugin>
            <groupId>com.heroku.sdk</groupId>
            <artifactId>heroku-maven-plugin</artifactId>
            <version>1.1.1</version>
        </plugin>
    </plugins>
</build>

我使用jetty-runner以WAR格式测试本地jetty服务器上的应用程序，并且每个GET请求都按预期工作。所以我使用Intellij IDE中的heroku:deploy-war将其部署到Heroku。部署成功，不显示错误/警告日志。之后，我导航到Heroku链接上的主页面（在页面上调用GET方法，不使用DL4J和ND4J），页面有效。但是当我尝试在使用DL4J和ND4J库的方法上调用GET请求时，浏览器页面会显示此错误，指出以下NoClassFound错误：

HTTP Status 500 - org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j

    type Exception report

    message org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j

    description The server encountered an internal error that prevented it from fulfilling this request.

    exception

    javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j
        org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
        org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)

    root cause

    org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j
        org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:278)
        org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:260)
        org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:509)
        org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334)
        org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
        org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
        org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
        org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
        org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)

    root cause

    java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j
        org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadTxt(WordVectorSerializer.java:1578)
        org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadTxtVectors(WordVectorSerializer.java:1502)
        com.example.MyResource.train(MyResource.java:61)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:606)
        org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
        org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
        org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
        org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
        org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
        org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
        org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
        org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
        org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
        org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
        org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
        org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
        org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
        org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)

    note The full stack trace of the root cause is available in the Apache Tomcat/8.0.30 logs.

更具体地说，只有当我从以下源代码调用train方法时才会发生这种情况：

package com.example;
import org.deeplearning4j.models.embeddings.loader.WordVectorSerializer;
import org.deeplearning4j.models.embeddings.wordvectors.WordVectors;

import javax.servlet.ServletContext;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;
import java.io.*;
import java.util.Arrays;
import java.util.Collection;

/**
 * Root resource (exposed at "myresource" path)
 */

@Path("test")
public class MyResource {

    WordVectors wordVectors=null;

    String loc="/WEB-INF/glove.6B.50d.txt";
    @javax.ws.rs.core.Context
    ServletContext context;


    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String getIt() {
        return  "Test";
     }

    @GET
    @Path("train")
    @Produces("text/plain")
    public String train(){


        String fullPath = context.getRealPath(loc);
        File file = new File(fullPath);
        try {
            wordVectors = WordVectorSerializer.loadTxtVectors(file);
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
        Collection<String> lst = wordVectors.wordsNearest("day", 20);


        return Arrays.toString(lst.toArray());
    }

}

启动Jetty嵌入式容器的类：

package com.example.heroku;

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.webapp.WebAppContext;

public class Main {

    public static void main(String[] args) throws Exception{
        String webPort = System.getenv("PORT");
        if (webPort == null || webPort.isEmpty()) {
            webPort = "8080";
        }

        final Server server = new Server(Integer.valueOf(webPort));
        final WebAppContext root = new WebAppContext();

        root.setContextPath("/");
        root.setParentLoaderPriority(true);

        final String webappDirLocation = "src/main/webapp/";
        root.setDescriptor(webappDirLocation + "/WEB-INF/web.xml");
        root.setResourceBase(webappDirLocation);

        server.setHandler(root);
        server.start();
        server.join();
    }
}

该错误似乎指出ND4J库中的某个类无法访问，尽管我已经检查了WEB-INF/lib文件夹，并且所有ND4J库都在那里。任何指导/帮助/提示将不胜感激。

Answer 1

听起来像标准的Servlet WebApp Classloader隔离正在发挥作用。

你必须知道你的类在哪里，是从服务器类路径加载，还是从WebApp的类路径加载。

当你用...时

java -cp target/classes:target/dependency/* com.example.heroku.Main

target/classes:target/dependency/*中定义的类和jar位于Server Classpath上，因此禁止WebApp查看来自Server Classpath的类。

然而......

您还声明root.setParentLoaderPriority(true);突破了Servlet WebApp Classloader隔离，但您现在在不同的地方拥有这些类的2个副本。一次从Server Classpath再次从WebApp的Classpath。如果它们的某些类是1类路径的一部分而某些类是另一类的一部分，那么有许多库将无法运行。

如果您打算使用此embedded-jetty方法，请考虑让WEB-INF/lib完全清空，或者不要在服务器类路径上声明这些类。

当你跑...

mvn jetty:run-war

然后这些类是WebApp类路径的一部分，WebApp可以看到它们，而服务器类路径没有它们的副本。

使用最新的Jetty和Jersey在上传的WAR文件中找不到依赖类到Heroku

1 个答案: