Question

My Spark Consumer因“日志记录”错误而失败。我发现浏览错误的原因是罐子不兼容。我使用Spark 1.6.3并且所有依赖项都在pom中使用，xml是1.6.3。我仍然得到同样的错误。下面是我的maven结构和执行时抛出的错误。

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:91)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:66)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:130)
    at org.apache.spark.streaming.kafka.KafkaUtils.createStream(KafkaUtils.scala)
    at datapipeline.SparkConsumer.main(SparkConsumer.java:60)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

我的pom.xml：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.iot.app.kafka</groupId>
    <artifactId>iotanalytics</artifactId>
    <!-- <packaging>jar</packaging> -->
    <version>1.0.0</version>
    <name>IoT Kafka Producer</name>


    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.0.0</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-network-common_2.10 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-network-common_2.10</artifactId>
            <version>1.6.3</version>
        </dependency>



        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10_2.11 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
            <version>2.0.0</version>
        </dependency>



        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.8.0</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.8.2</version>
        </dependency>

         <dependency> 
         <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-scala_2.10</artifactId> 
        <version>2.8.2</version>
         </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.8.2</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.8.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/javax.json/javax.json-api -->
        <dependency>
            <groupId>javax.json</groupId>
            <artifactId>javax.json-api</artifactId>
            <version>1.1</version>
        </dependency>



        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.11 -->

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>1.6.3</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.glassfish</groupId>
            <artifactId>javax.json</artifactId>
            <version>1.0.4</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.2.0</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>1.6.3</version>
            <scope>provided</scope>
        </dependency>




        <dependency>
            <groupId>org.spark-project.spark</groupId>
            <artifactId>unused</artifactId>
            <version>1.0.0</version>
            <scope>provided</scope>
        </dependency>




        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>


        <!-- <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> 
            <version>2.11.8</version> </dependency> -->


        <!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.9</version>
        </dependency>


        <dependency>
            <groupId>com.typesafe.akka</groupId>
            <artifactId>akka-actor_2.11</artifactId>
            <version>2.3.11</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-8-assembly_2.11 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8-assembly_2.11</artifactId>
    <version>2.0.0</version>
</dependency>



        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>1.6.3</version>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.4</version>
            <scope>test</scope>
        </dependency>



    </dependencies>


    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>

    </properties>

</project>

Answer 1

有一些罐子存在火花的较低版本，我以前建造过。那些没有被冲走。我删除了旧版本的jar并查看了我的pom.xml，并确保所有依赖项具有相同的版本并构建项目。程序运行成功。这是我的pom.xml

<activity
    android:name=".activity.UserProfileActivity"
    android:configChanges="keyboardHidden|orientation|screenSize"
    android:label="@string/app_name"
    android:launchMode="singleTask"
    android:parentActivityName=".activity.UserProfileActivity"
    android:screenOrientation="portrait"
    tools:ignore="UnusedAttribute">
    <intent-filter>
        <category android:name="android.intent.category.DEFAULT" />
        <action android:name="android.intent.action.VIEW" />
        <data android:scheme="com.test.android.activity.UserProfileActivity" />
    </intent-filter>
</activity>

Answer 2

类org.apache.spark.Logging在Spark版本1.5.2或更低版本中可用，而不是在更高版本中。

尝试将Spark版本降级为1.5.2

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>1.5.2</version>
    <scope>provided</scope>
</dependency>

线程“main”中的异常java.lang.NoClassDefFoundError：org / apache / spark / internal / Logging

2 个答案: