Question

使用spark-testing-base进行单元测试时我收到以下错误

java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/MRVersion
at org.apache.hadoop.hive.shims.Hadoop23Shims.isMR2(Hadoop23Shims.java:852)
at org.apache.hadoop.hive.shims.Hadoop23Shims.getHadoopConfNames(Hadoop23Shims.java:923)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:358)
at com.holdenkarau.spark.testing.DataFrameSuiteBaseLike$class.newTemporaryConfiguration$1(DataFrameSuiteBase.scala:63)

我正在使用以下依赖项。这是从intellj开始的，但是当我执行sbt test时它失败了

 libraryDependencies ++= Seq(
  "org.scalaj" %% "scalaj-http" % "2.3.0",
  "org.apache.spark" %% "spark-core" % "1.6.0-cdh5.8.4" % "provided",
  "org.apache.spark" %% "spark-hive" % "1.6.0-cdh5.8.4" % "provided",
  "org.apache.spark" %% "spark-sql" % "1.6.0-cdh5.8.4" % "provided",
  "com.holdenkarau" %% "spark-testing-base" % "1.6.0_0.8.0" % Test,
  "org.scalatest" %% "scalatest" % "2.2.6" % "test",
  "org.mockito" % "mockito-core" % "2.8.9" % "test",
  "org.flywaydb" % "flyway-core" % "5.0.2" % "test"
)

libraryDependencies ++= Seq(
  "org.apache.hive" % "hive-jdbc" % "1.1.0-cdh5.8.4" % "provided",
  "org.apache.hive" % "hive-exec" % "1.1.0-cdh5.8.4" % "provided",
  "org.apache.hive" % "hive-metastore" % "1.1.0-cdh5.8.4" % "provided",
  "org.apache.hive" % "hive-shims" % "1.1.0-cdh5.8.4" % "provided",
  "org.apache.hive.shims" % "hive-shims-common" % "1.1.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-common" % "2.6.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-hdfs" % "2.6.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.6.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-mapreduce-client-common" % "2.6.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-mapreduce-client-jobclient" % "2.6.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-yarn-api" % "2.6.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-yarn-common" % "2.6.0-cdh5.8.4" % "provided",
  "org.apache.hadoop" % "hadoop-yarn-client" % "2.6.0-cdh5.8.4" % "provided",
  "org.spark-project.akka" %% "akka-actor" % "2.2.3-shaded-protobuf" % "provided"
  )

Answer 1

已解决此...使用以下依赖项正在修复此问题。

“org.apache.hadoop”％“hadoop-core”％“2.6.0-mr1-cdh5.8.4”％“提供”

使用spark-testing-base时，NoClassDefFoundError：org / apache / hadoop / mapred / MRVersion

1 个答案: