Spark Streaming App通过代码提交

时间：2014-11-27 05:19:30

标签： java maven hadoop apache-spark spark-streaming

我正在尝试通过代码提交spark streaming应用程序

SparkConf sparkConf= new SparkConf();
JavaStreamingContext jssc = new JavaStreamingContext(master, appName, new Duration(60*1000), sparkHome, sparkJar);

给了绝对路径dor SparkJar和sparkHome master spark://xyz:7077

我尝试以相同的方式提交批处理，但它工作但不适用于流式处理 我得到以下错误.. 

14/11/26 17:42:25 INFO spark.HttpFileServer: HTTP File server directory is /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-cd7b30cd-cf95-4e52-8eb4-1c1dccc2d58f
14/11/26 17:42:25 INFO spark.HttpServer: Starting HTTP Server
14/11/26 17:42:25 INFO server.Server: jetty-8.1.14.v20131031
14/11/26 17:42:25 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:50016
14/11/26 17:42:25 INFO server.Server: jetty-8.1.14.v20131031
14/11/26 17:42:25 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
14/11/26 17:42:25 INFO ui.SparkUI: Started SparkUI at http://xxx.xx.xxx.xx:4040
14/11/26 17:42:30 INFO spark.SparkContext: Added JAR /Volumes/Official/workspace/ZBI/target/ZBI-0.0.1-SNAPSHOT-jar-with-dependencies.jar at http://xxx.xx.xxx.xx:50016/jars/ZBI-0.0.1-SNAPSHOT-jar-with-dependencies.jar with timestamp 1417003949988
Exception in thread "main" java.lang.NoClassDefFoundError: **org/apache/spark/ui/SparkUITab**
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

我正在使用maven，以下是我的pom.xml 

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>BetaTestTool</groupId>
    <artifactId>TestTool</artifactId>
    <packaging>jar</packaging>
    <version>0.0.1-SNAPSHOT</version>
    <description></description>
    <build>
                <plugins>
                        <plugin>
                                <artifactId>maven-compiler-plugin</artifactId>
                                <version>3.1</version>
                                <configuration>
                                        <source>1.5</source>
                                        <target>1.5</target>
                                </configuration>
                        </plugin>
            <plugin>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.0.1</version>
            </plugin>
              <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>2.3</version>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id> <!-- this is used for inheritance merges -->
            <phase>package</phase> <!-- bind to the packaging phase -->
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
        </plugins>
    </build>
    <dependencies>
        **<dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>servlet-api</artifactId>
            <version>2.5</version>
        </dependency>**  
        <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.0.2</version>
    </dependency>
    <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.10</artifactId>
      <version>1.0.2</version>
    </dependency>
   <!-- <dependency> Spark dependency
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.10</artifactId>
      <version>1.0.2</version>
    </dependency> -->

  <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
      <version>1.7.5</version>
    </dependency>

    </dependencies>
</project>

我在异常

之后遇到此异常 
14/11/27 10:43:13 INFO spark.HttpFileServer: HTTP File server directory is /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-b162a8c1-0d77-48db-b559-2b242449db3e
14/11/27 10:43:13 INFO spark.HttpServer: Starting HTTP Server
14/11/27 10:43:13 INFO server.Server: jetty-8.1.14.v20131031
14/11/27 10:43:13 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:62675
Exception in thread "main" java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package
    at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
    at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

然后我评论了javax.servlet依赖...之后我得到了第一个提到的错误..还请建议如何排除depandancy ..我尝试将范围作为编译并提供但是没有工作.. < / p>

非常感谢任何帮助

我的Pom Tree如下

--- maven-dependency-plugin:2.8:tree (default-cli) @ ZBI ---
[INFO] BetaBI:ZBI:jar:0.0.1-SNAPSHOT
[INFO] \- org.apache.spark:spark-core_2.10:jar:1.0.2:compile
[INFO]    \- org.apache.hadoop:hadoop-client:jar:1.0.4:compile
[INFO]       \- org.apache.hadoop:hadoop-core:jar:1.0.4:compile
[INFO]          \- commons-configuration:commons-configuration:jar:1.6:compile
[INFO]             \- commons-collections:commons-collections:jar:3.2.1:compile

如何在核心spark中的hadoop依赖中排除javax.servlet？

2 个答案:

答案 0 :(得分：0)

pom.xml中似乎缺少Spark流依赖性。

<dependency> <!-- Spark streaming dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.10</artifactId>
  <version>1.0.2</version>
</dependency>

答案 1 :(得分：0)

我认为这是版本问题，因为spark 1.1.x导入了一个名为SparkUITab的新类，它从WebUITab扩展，但在spark 1.0.2中只使用WebUITab，所以如果你的spark-core_2.10的版本是1.0 .2但是你的spark-streaming-kafka_2.10的版本是1.1.0，它会导致版本匹配问题。所以你最好确保使用相同的版本。

相关问题
最新问题