我在创建JavaStreamingContext时遇到AbstractMethodError异常。 我的依赖项pom如下;无法找到线索,任何人都可以建议这里出了什么问题吗?
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.2</version>
</dependency>
线程“ main”中的异常java.lang.AbstractMethodError 在org.apache.spark.util.ListenerBus $ class。$ init $(ListenerBus.scala:35) 在org.apache.spark.streaming.scheduler.StreamingListenerBus。(StreamingListenerBus.scala:30) 在org.apache.spark.streaming.scheduler.JobScheduler。(JobScheduler.scala:57) 在org.apache.spark.streaming.StreamingContext。(StreamingContext.scala:184) 在org.apache.spark.streaming.StreamingContext。(StreamingContext.scala:76) 在org.apache.spark.streaming.api.java.JavaStreamingContext。(JavaStreamingContext.scala:130)
答案 0 :(得分:1)
您在此处混合了许多版本的Spark
首先,如果您使用的是Apache Spark 2.3.1和kafka 0.10 +
我建议以下内容:
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<!-- Keep the same Spark version as before -->
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<!-- Keep the same Spark version as before -->
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<!-- Support for kafka version 0.8 is deprecated as of Spark 2.3.1 and you add the dependencies for kafka 0.10+ above -->
<version><your_kafka_version_0.10+></version>
</dependency>
很高兴知道您如何构建/部署应用程序?根据您的运行时环境,您可能想要添加一些提供的作用域,以防止生成的程序包与现有环境之间发生冲突。
希望有帮助