Question

我正在尝试运行一个由Spark结构化流组成的应用程序-来自kafka的数据输入。 Spark版本是2.4.0，scala版本是2.12.7。我正在使用sbt制作多个胖子-我的项目是多模块项目。建立罐子不是问题。当我尝试用罐子spark-submit时，发生了NoSuchMethodError。

我从provided移除了spark-sql-kafka-0-10范围。

val sparkSqlKafka = "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion

在assemblyMergeStrategy中，我在下面添加了一行。

case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => MergeStrategy.concat

这是完整的错误日志。



2019-01-08 11:55:12 ERROR ApplicationMaster:91 - User class threw exception: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.kafka010.KafkaSourceProvider could not be instantiated
java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.kafka010.KafkaSourceProvider could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
    at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
    at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)
    at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:161)
    at ThisIsMyClass$.main(ThisIsMyClass.scala:28)
    at ThisIsMyClass.main(ThisIsMyClass.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:678)
Caused by: java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.$init$(Lorg/apache/spark/internal/Logging;)V
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.<init>(KafkaSourceProvider.scala:44)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at java.lang.Class.newInstance(Class.java:442)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 19 more

2019-01-08 11:55:12 ERROR ApplicationMaster:91 - User class threw exception: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.kafka010.KafkaSourceProvider could not be instantiated java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.kafka010.KafkaSourceProvider could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247) at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259) at scala.collection.AbstractTraversable.filter(Traversable.scala:104) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630) at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:161) at ThisIsMyClass$.main(ThisIsMyClass.scala:28) at ThisIsMyClass.main(ThisIsMyClass.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:678) Caused by: java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.$init$(Lorg/apache/spark/internal/Logging;)V at org.apache.spark.sql.kafka010.KafkaSourceProvider.<init>(KafkaSourceProvider.scala:44) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 19 more

编辑1。

下面的整个依赖项。

val sparkVersion = "2.4.0"
val typesafeConfigVersion = "1.3.3"
val scalaTestVersion = "3.0.5"
val junitVersion = "4.12"
val sparkCore = "org.apache.spark" %% "spark-core" % sparkVersion % "provided"
val sparkSql = "org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
val sparkMllib = "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided"
val typesafeConfig = "com.typesafe" % "config" % typesafeConfigVersion
val scalaTest = "org.scalatest" %% "scalatest" % scalaTestVersion % Test
val junit = "junit" % "junit" % "4.12" % Test
val logback = "ch.qos.logback" % "logback-classic" % "1.2.3"
val scalaLogging = "com.typesafe.scala-logging" %% "scala-logging" % "3.9.0"
val sparkStreaming = "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
val sparkSqlKafka = "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion

编辑2。

我的val sparkVersion = "2.4.0" val typesafeConfigVersion = "1.3.3" val scalaTestVersion = "3.0.5" val junitVersion = "4.12" val sparkCore = "org.apache.spark" %% "spark-core" % sparkVersion % "provided" val sparkSql = "org.apache.spark" %% "spark-sql" % sparkVersion % "provided" val sparkMllib = "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided" val typesafeConfig = "com.typesafe" % "config" % typesafeConfigVersion val scalaTest = "org.scalatest" %% "scalatest" % scalaTestVersion % Test val junit = "junit" % "junit" % "4.12" % Test val logback = "ch.qos.logback" % "logback-classic" % "1.2.3" val scalaLogging = "com.typesafe.scala-logging" %% "scala-logging" % "3.9.0" val sparkStreaming = "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided" val sparkSqlKafka = "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion发现了一些依赖版本问题。

因此，我已将构建配置更改为仅使用的一个版本-具有slf4j-api依赖性的版本匹配。并排除其他slf4j-api。

仍然是相同的错误。：（

编辑3。

我在spark-core脚本中添加了slf4j-api。

仍然是相同的错误。

Answer 1

问题解决了。

打开spark-shell时，发现spark版本为 2.4.0 ，scala版本为 2.11 。
我在build.sbt中的Scala版本是 2.12 。

Scala版本是关键！

感谢大家。

Answer 2

请更新您的依赖项。

    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.7</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.7</version>
    </dependency>

NoSuchMethodError：org.apache.spark.internal.Logging

2 个答案: