Question

我有一个非常基本和简单的应用程序，其中有一个topic1和topic2，一个从stream1读取的spark流应用程序，以及另一个将结果发送到topic2的应用程序。

阅读：

import org.apache.spark.sql.{DataFrame, SparkSession}


object ReadKafkaTopic {
  def readStream(spark: SparkSession, brokers: String, topic: String): DataFrame = {
    spark
      .readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", brokers)
      .option("failOnDataLoss", false)
      .option("subscribe", topic)
      .option("startingOffsets", "latest")
      .load()
  }
}

写：

import org.apache.spark.sql.{DataFrame}


object WriteKafkaTopic {
  def writeStream(df: DataFrame, brokers: String, topic: String): Unit = {
    df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
      .writeStream
      .format("kafka")
      .option("kafka.bootstrap.servers", brokers)
      .option("topic", topic)
      .option("checkpointLocation", "/tmp/checkpoint")
      .start()
  }
}

这很棒。现在，我尝试通过本地卡夫卡“测试”它来进行测试（使用 net.manub.embeddedkafka.EmbeddedKafka）

当我运行测试时，出现此错误：

Cause: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.8

因此，在网上阅读类似的问题后，我将所有的杰克逊依赖项都传递给了2.6.7（当然，在尝试了不同的选择之后）：

scalaVersion := "2.12.8"
val sparkVersion = "2.4.2"

// https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.2.0"

// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion

// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion

// https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10
libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion % "provided"

// testing dependencies
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % "test"

libraryDependencies += "io.github.embeddedkafka" %% "embedded-kafka" % "2.2.0" % "test"

// https://mvnrepository.com/artifact/org.apache.curator/curator-test
libraryDependencies += "org.apache.curator" % "curator-test" % "4.2.0" % Test

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.6.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.6.7"
//
//// https://mvnrepository.com/artifact/com.fasterxml.jackson.datatype/jackson-datatype-jdk8
dependencyOverrides += "com.fasterxml.jackson.datatype" % "jackson-datatype-jdk8" % "2.6.7"

但是我继续在相同的错误上运行不兼容的杰克逊版本：2.9.8

有人遇到这个问题并且知道如何解决吗？

火花流-kafka fastxml.jackson为2.6，但不兼容的Jackson版本：2.9.8

0 个答案: