无法使用kafka-storm

时间:2016-04-04 12:29:52

标签: java maven apache-kafka apache-storm

我向Apache Storm(版本0.10.0.2.4.0.0-169)提交我的拓扑时遇到错误,该版本使用来自storm-kafka的Kafka spout。我正在使用github的最新版本的storm-kafka,并且正在运行kafka 0.9.0.2.4。

我可以使用maven下载的jar文件为storm-core和storm-kafka运行这个拓扑的相当普通版本,但是我需要对spout类进行更改,所以我必须从源代码下载并自己构建它,它在maven中编译我的ubuntu开发环境,但是在向风暴提交拓扑时我收到错误。

“找到多个defaults.yaml资源。您可能正在使用拓扑jar构建Storm jar”

我认为这是因为我在我的maven依赖项中包含了storm-core(我从源代码编译的本地版本(并添加到我的本地maven资源库),因为在storm-kafka中需要它,而从mvn中提取的版本是最新版本的storm-kafka所需缺少的类)但是,如果我将storm-core设置为pom.xml中提供的,我在提交时会收到以下错误 -

“java.lang.class.NoClassDefFoundError:org / apache / storm / spout / scheme”

我认为是因为它试图使用一些缺少一些最新文件的风暴核心版本。

我似乎陷入了一个循环,为了编译最新版本的storm-kafka,我需要最新版本的storm-core,需要从源代码自编译,但我似乎无法将此部署到服务器而不会导致冲突。

我是否需要在风暴服务器上更新该jar?我担心的是,我最终需要更新它的依赖项等。

下面是我的pom.xml文件

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.automatedintelligence.datastorm</groupId>
<artifactId>DataStorm</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>DataStorm</name>
<url>https://github.com/markmcgookin/DataStorm</url>
<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
    <repository>
            <id>local-maven-repo</id>
            <url>file:///${project.basedir}/src/main/java/dependency/repo</url>
        </repository>
    <repository>
        <id>central</id>
        <url>http://central.maven.org/maven2/</url>
    </repository>
    <repository>
        <id>github-releases</id>
        <url>http://oss.sonatype.org/content/repositories/github-releases/</url>
    </repository>
    <repository>
        <id>mvnrepository</id>
        <url>http://mvnrepository.com/</url>
    </repository>
    <repository>
        <id>clojars.org</id>
        <url>http://clojars.org/repo</url>
    </repository>
    <repository>
        <id>twitter4j</id>
        <url>http://twitter4j.org/maven2</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>3.8.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-collections4</artifactId>
        <version>4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>1.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>                                                                                                                                       
        <groupId>org.apache.hadoop</groupId>                                                                                                           
        <artifactId>hadoop-common</artifactId>                                                                                                         
        <version>2.7.2</version>   
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>                                                                                         
    </dependency> 
    <dependency>
        <groupId>storm</groupId>
        <artifactId>storm</artifactId>
        <version>0.9.0-rc2</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>0.9.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
        <version>2.0.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>
<build>
    <plugins>
        <!-- bind the maven-assembly-plugin to the package phase
        this will create a jar file without the Storm
        dependencies suitable for deployment to a cluster.-->
        <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifest>
                        <mainClass/>
                    </manifest>
                </archive>
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>com.theoryinpractise</groupId>
            <artifactId>clojure-maven-plugin</artifactId>
            <version>1.3.8</version>
            <extensions>true</extensions>
            <configuration>
                <sourceDirectories>
                    <sourceDirectory>src/clj</sourceDirectory>
                </sourceDirectories>
            </configuration>
            <executions>
                <execution>
                    <id>compile</id>
                    <phase>compile</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
                <execution>
                    <id>test</id>
                    <phase>test</phase>
                    <goals>
                        <goal>test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.6</source>
                <target>1.6</target>
                <excludes>
                    <exclude>sl4j:log4j:log4j:jar:</exclude>
                </excludes>
            </configuration>
        </plugin>
    </plugins>
</build>

0 个答案:

没有答案