单个JVM中的多个Spark服务器

时间:2017-01-03 20:52:27

标签: java spark-java

有没有办法在同一个JVM中运行多个Sparkjava服务器实例?我在“插件”软件中使用它并根据外部情况我的插件的多个实例可能会启动然后导致

java.lang.IllegalStateException: This must be done before route mapping has begun
at spark.SparkBase.throwBeforeRouteMappingException(SparkBase.java:256)
at spark.SparkBase.port(SparkBase.java:101)
at com.foo.bar.a(SourceFile:59)

在我看来,通过查看它是围绕代码中的静态字段构建的代码,所以我正在考虑类加载器技巧或使用SparkServerFactory以某种方式消除SparkBase

3 个答案:

答案 0 :(得分:9)

从Spark 2.5,您可以使用ignite()

http://sparkjava.com/news.html#spark25released

示例:

public static void main(String[] args) {
    igniteFirstSpark();
    igniteSecondSpark();
}

static void igniteSecondSpark() {
    Service http = ignite();

    http.get("/basicHello", (q, a) -> "Hello from port 4567!");
}

static void igniteFirstSpark() {
    Service http = ignite()
                      .port(8080)
                      .threadPool(20);

    http.get("/configuredHello", (q, a) -> "Hello from port 8080!");
}

我个人初始化它们是这样的:

import spark.Service

public static void main(String[] args) {
    Service service1 = Service.ignite().port(8080).threadPool(20)
    Service service2 = Service.ignite().port(8081).threadPool(10)
}

我建议阅读how to use those services outside your main method,我认为这在这里很有用。

答案 1 :(得分:1)

诀窍是忽略spark.Spark中实现的Spark周围的外部静态shell,并直接使用内部spark.webserver.SparkServer。代码中存在一些需要解决方法的障碍,例如: spark.webserver.JettyHandler不公开,因此您无法从代码中对其进行实例化,但您可以将自己的类放在该包中并将其公开。

所以解决方案就是这样:

SimpleRouteMatcher routeMatcher1 = new SimpleRouteMatcher();
routeMatcher1.parseValidateAddRoute("get '/foo'", "*/*", wrap("/foo", "*/*", (req, res) -> "Hello World 1"));

MatcherFilter matcherFilter1 = new MatcherFilter(routeMatcher1, false, false);
matcherFilter1.init(null);
PublicJettyHandler handler1 = new PublicJettyHandler(matcherFilter1);
SparkServer server1 = new SparkServer(handler1);

new Thread(() -> {
            server1.ignite("0.0.0.0", 4567, null, null, null, null, "/META-INF/resources/", null, new CountDownLatch(1),
                    -1, -1, -1);
        }).start();

需要在代码库中复制wrap方法:

protected RouteImpl wrap(final String path, String acceptType, final Route route) {
        if (acceptType == null) {
            acceptType = "*/*";
        }
        RouteImpl impl = new RouteImpl(path, acceptType) {
            @Override
            public Object handle(Request request, Response response) throws Exception {
                return route.handle(request, response);
            }
        };
        return impl;
    }

如果您的应用中需要多个Spark服务器,这似乎是一种可行的解决方法。

答案 2 :(得分:0)

我有这个问题用Spark运行单元测试,修复它我修改了pom.xml文件。

forkCount=1 reuseForks=false

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>${surefire.version}</version>
            <dependencies>
                <dependency>
                    <groupId>org.junit.platform</groupId>
                    <artifactId>junit-platform-surefire-provider</artifactId>
                    <version>${junit.platform.version}</version>
                </dependency>
            </dependencies>
            <configuration>
                <forkCount>1</forkCount>
                <reuseForks>false</reuseForks>