Question

我想通过REPL查看我的scala spark代码这段代码适用于spark-shell，但本地scala与sbt没有用（在命令行中执行sbt后，执行console命令启动REPL。）

我该怎么办？

码

import org.apache.spark.SparkContext  
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

val conf = new SparkConf().setAppName("RddSample").setMaster("local[*]")
val sc = new SparkContext(conf)
val data = (0 to 10).toArray
val inputRDD = sc.parallelize(data)

错误

com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.9.4 requires Jackson Databind version >= 2.9.0 and < 2.10.0 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
  at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:18)
  at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:692)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:821)
... 40 elided

build.sbt

name := "Sratup"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "2.3.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % sparkVersion
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.4"
libraryDependencies +="com.fasterxml.jackson.core" % "jackson-databind" % "2.9.4"
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.4"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.4"

Answer 1

Spark版本（事实上，AFAIK，每个版本从2.1.0开始）已包含com.fasterxml.jackson.core，因此您无需将它们放在build.sbt中。

如果您想使用其他版本，请将其从libraryDependencies中删除，并将其包含在dependencyOverrides

中

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala" % "2.9.4"

您可以查看spark 2.3.0包含here

的依赖项

使用本地REPL检查Spark RDD的代码

1 个答案: