Question

我的系统中没有hadoop安装，但是我有winutils 并且在本地运行spark蜂巢时出现此错误：-

Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:869)
at com.sebi.count.ScalaSix$.main(ScalaSix.scala:14)
at com.sebi.count.ScalaSix.main(ScalaSix.scala)

我的代码是：-

package com.sebi.count
import org.apache.spark.sql.SparkSession
object ScalaSix {
  def main(args: Array[String]): Unit = {
    System.setProperty("hadoop.home.dir", "C:/winutils")
    val warehouseLocation = "spark-warehouse"
    val spark = SparkSession.builder().appName("Spark Hive Example")
        .config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate()
    spark.sparkContext.setLogLevel("ERROR")
val df = spark.sql("CREATE TABLE IF NOT EXITS NAME(Id Int, Name String, Age Int,Salary Int)")
    val df1  = spark.sql("LOAD DATA LOCAL INPATH 'C:/Users/rittde/IdeaProjects/untitled/src/main/resources' " +
      "into table Name")
    val df2 = spark.sql("select * from Name")
    df2.show()
}

我的sbt依赖项是：-

   name := "untitled"
    version := "0.1"

scalaVersion := "2.11.12"
    libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0" libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.0" %"provided"

能否请您建议如何解决该错误。预先感谢。

Answer 1

我有同样的错误。我通过更改解决了

libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.3" % "provided"

到

libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.3"

就我而言。这是一个说明sbt % "provide"配置的链接。 SBT assembly jar exclusion

无法通过Hive支持实例化SparkSession，因为未找到Hive类

1 个答案: