Question

从KafkaStreams库从2.3升级到2.5并保持相同的Scala版本时，运行时会出现以下错误，从而中断：

[info] com.mypackage.dp.streams.applications.EmbeddedKafkaVisitorTest *** ABORTED ***
[info]   java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
[info]   at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)
[info]   at kafka.api.ApiVersion.compare(ApiVersion.scala:139)
[info]   at kafka.api.ApiVersion.compare$(ApiVersion.scala:138)
[info]   at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
[info]   at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
[info]   at scala.math.Ordered.$greater$eq(Ordered.scala:88)
[info]   at scala.math.Ordered.$greater$eq$(Ordered.scala:88)
[info]   at kafka.api.KAFKA_2_5_IV0$.$greater$eq(ApiVersion.scala:339)
[info]   at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1529)
[info]   at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1237)
[info]   ...
[info]   Cause: java.lang.ClassNotFoundException: scala.math.Ordering$$anon$7
[info]   at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
[info]   at sbt.internal.ManagedClassLoader.findClass(ManagedClassLoader.java:98)
[info]   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
[info]   at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
[info]   at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)
[info]   at kafka.api.ApiVersion.compare(ApiVersion.scala:139)
[info]   at kafka.api.ApiVersion.compare$(ApiVersion.scala:138)
[info]   at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
[info]   at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
[info]   at scala.math.Ordered.$greater$eq(Ordered.scala:88)
[info]   ...
[error] Test suite com.mypackage.dp.streams.applications.EmbeddedKafkaVisitorTest failed with java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7.
[error] This may be due to the ClassLoaderLayeringStrategy (ScalaLibrary) used by your task.
[error] To improve performance and reduce memory, sbt attempts to cache the class loaders used to load the project dependencies.
[error] The project class files are loaded in a separate class loader that is created for each test run.
[error] The test class loader accesses the project dependency classes using the cached project dependency classloader.
[error] With this approach, class loading may fail under the following conditions:
[error] 
[error]  * Dependencies use reflection to access classes in your project's classpath.
[error]    Java serialization/deserialization may cause this.
[error]  * An open package is accessed across layers. If the project's classes access or extend
[error]    jvm package private classes defined in a project dependency, it may cause an IllegalAccessError
[error]    because the jvm enforces package private at the classloader level.
[error] 
[error] These issues, along with others that were not enumerated above, may be resolved by changing the class loader layering strategy.
[error] The Flat and ScalaLibrary strategies bundle the full project classpath in the same class loader.
[error] To use one of these strategies, set the  ClassLoaderLayeringStrategy key
[error] in your configuration, for example:
[error] 
[error] set mypackage / Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
[error] set mypackage / Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
[error] 
[error] See ClassLoaderLayeringStrategy.scala for the full list of options.

尝试了建议的策略，但有相同的错误。

我的build.sbt依赖性和策略：

scalaVersion := "2.12.4"
val kafka_streams_scala_version = "2.5.0"
val kafka_version = "2.5.0"
libraryDependencies ++= {
  Seq(
    "org.apache.kafka" % "kafka-streams" % kafka_version,
    "org.apache.kafka" %% "kafka" % kafka_version,
    "org.apache.kafka" % "kafka-tools" % kafka_version,
    "org.apache.kafka" %% "kafka-streams-scala" % kafka_version,
    "org.apache.kafka" % "kafka-clients" % kafka_version,
    "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.2",
    "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.2",
    "io.spray" %% "spray-json" % "1.3.4",

    // testing
    "org.scalatest" %% "scalatest" % "3.0.5" % "test",
    "org.scalactic" %% "scalactic" % "3.0.5",
    "net.manub" %% "scalatest-embedded-kafka-streams" % "2.0.0" % "test",
    "org.apache.kafka" % "kafka-streams-test-utils" % kafka_version % "test",

    // Mongo
    "org.mongodb.scala" %% "mongo-scala-driver" % "2.2.1",

    // logging
    "org.slf4j" % "slf4j-api" % "1.7.25",
    "com.typesafe.scala-logging" %% "scala-logging" % "3.9.2",

    // Monitoring
    "com.datadoghq" % "java-dogstatsd-client" % "2.7",

    // misc
    "commons-httpclient" % "commons-httpclient" % "3.1",
    "org.apache.commons" % "commons-lang3" % "3.7",
    "commons-io" % "commons-io" % "2.6",
    "commons-codec" % "commons-codec" % "1.11",
    "commons-validator" % "commons-validator" % "1.6",
    "com.jsuereth" %% "scala-arm" % "2.0",
    "org.rogach" %% "scallop" % "3.1.2",
    "com.github.nikita-volkov" % "sext" % "0.2.4",
    "com.github.x25" % "ip-subnet-tree" % "1.0.3",
    "com.sauldhernandez" %% "autoschema" % "1.0.4",

    // from quasar
    "com.netaporter" %% "scala-uri" % "0.4.16",
    "org.yaml" % "snakeyaml" % "1.19",
    "com.typesafe" % "config" % "1.3.3",
    "joda-time" % "joda-time" % "2.9.9",

    "jakarta.ws.rs" % "jakarta.ws.rs-api" % "2.1.4",   
    "io.sentry" % "sentry" % "1.7.30",
    "io.sentry" % "sentry-logback" % "1.7.30"
  ).map(_ exclude("javax.ws.rs", "javax.ws.rs-api"))

}

fork in run := true
Compile / compileOrder := CompileOrder.Mixed

Test / parallelExecution := false
parallelExecution in ThisBuild := false

// META-INF discarding
assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

什么可能导致此问题？如何解决编译问题？

Answer 1

我只需要将Scala版本升级到“编译的依赖项”部分here中发布的版本，即：

scalaVersion := "2.12.10"

也可以升级到2.13，但这导致我在项目中的许多其他依赖关系中断。

Kafka Streams 2.3到2.5升级打破了Scala的编译

1 个答案: