Question

I'm trying to create my custom spark processor in springxd 1.2, and when I'm creating the module I got the following error.

java.lang.IllegalStateException: Either java or scala module should be present

Which module does it mean. Is it about a non imported library?

My module definition is:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns:int="http://www.springframework.org/schema/integration"
   xsi:schemaLocation="http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">


<int:channel id="input"/>
<int:channel id="output"/>
<int:transformer input-channel="input" output-channel="output">
    <bean class="spark.MetricKeyAggregator"/>
</int:transformer>

My pom is this one:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
    <groupId>org.springframework.xd</groupId>
    <artifactId>spring-xd-module-parent</artifactId>
    <version>1.2.0.RELEASE</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>metric-aggregator-spark</artifactId>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.2.1</version>
    </dependency>
    <dependency>
        <groupId>es.onebox</groupId>
        <artifactId>metric-aggregator-domain</artifactId>
        <version>1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.10</artifactId>
        <version>1.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.xd</groupId>
        <artifactId>spring-xd-spark-streaming</artifactId>
        <version>1.2.0.RELEASE</version>
    </dependency>
</dependencies>

<repositories>
    <repository>
        <id>spring-io-release</id>
        <url>http://repo.spring.io/release</url>
    </repository>
    <repository>
        <id>spring-io-snapshot</id>
        <url>https://repo.spring.io/libs-snapshot</url>
    </repository>
    <repository>
        <id>paho-releases</id>
        <url>https://repo.eclipse.org/content/repositories/paho-releases</url>
    </repository>
</repositories>

The stack strace is the following:

DeploymentStatus{state=failed,error(s)=java.lang.IllegalStateException: Either java or scala module should be present.
at org.springframework.xd.dirt.plugins.spark.streaming.SparkStreamingPlugin.postProcessModule(SparkStreamingPlugin.java:145)
at org.springframework.xd.dirt.module.ModuleDeployer.postProcessModule(ModuleDeployer.java:238)
at org.springframework.xd.dirt.module.ModuleDeployer.doDeploy(ModuleDeployer.java:218)
at org.springframework.xd.dirt.module.ModuleDeployer.deploy(ModuleDeployer.java:200)
at org.springframework.xd.dirt.server.container.DeploymentListener.deployModule(DeploymentListener.java:365)
at org.springframework.xd.dirt.server.container.DeploymentListener.deployStreamModule(DeploymentListener.java:334)
at org.springframework.xd.dirt.server.container.DeploymentListener.onChildAdded(DeploymentListener.java:181)
at org.springframework.xd.dirt.server.container.DeploymentListener.childEvent(DeploymentListener.java:149)
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509)
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503)
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:500)
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
at org.apache.curator.framework.recipes.cache.PathChildrenCache$10.run(PathChildrenCache.java:762)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Answer 1

模块中没有Processor。

您的MetricKeyAggregator需要实现Processor并简单地定义为bean;你不需要输入/输出通道或变压器。

有关示例，请参阅this test。

Error while creating spark processor module in springxd : Either java or scala module should be present

1 个答案: