Question

将MIN(10, SUMIFS($AI$42:$AI$51;$AH$42:$AH$51;"<"&AH51)+1)与spark structured streaming一起使用时，我看到了spark-sql-kafka-0-10_2.11的。基于另一个问题Cannot run queries in SQLContext from Apache Spark SQL 1.5.2, getting java.lang.NoSuchMethodError 我试图明确设置 jackson 版本。

已经尝试了版本2.9.6、2.4.3、2.9.0。 MethodNotFoundError说“杰克逊版本太旧”。其他版本说

原因：com.fasterxml.jackson.databind.JsonMappingException：不兼容的杰克逊版本：2.9.0

这是2.4.3的完整ST：

2.9.0

还要注意，我在19/05/10 11:30:18 ERROR MicroBatchExecution: Query [id = dbd581ba-42d7-4496-9fde-fe04dab6e7b4, runId = b5b023df-cb39-4048-90dc-e9a57cce4883] terminated with error java.lang.ExceptionInInitializerError at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:247org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3365) at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3364) at org.apache.spark.sql.Dataset.collect(Dataset.scala:2783) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537) at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) at at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351) at .. org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751) at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82) at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)中确实有排除项：

pom.xml

AWS的类似排除方式

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>${jackson.databind.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
       <exclusions>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.fasterxml.jackson.core</groupId>
                <artifactId>jackson-databind</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

有什么想法可以解决这里的杰克逊版本问题？

Answer 1

通过在$ SPARK_HOME / jars目录中查找并搜索jackson-databind找到了答案：

$ll *jackson-databind*
-rw-r--r--@ 1 steve  staff  1165323 Mar 26 17:13 jackson-databind-2.6.7.1.jar

因此，为

更新pom.xml

    <jackson.databind.version>2.6.7</jackson.databind.version>

解决了这个问题。

Answer 2

对于Scala，添加您的build.sbt文件：

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7"

Spark结构化流中的Jackson版本问题

2 个答案: