我的系统中没有hadoop安装,但是我有winutils 并且在本地运行spark蜂巢时出现此错误:-
Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:869)
at com.sebi.count.ScalaSix$.main(ScalaSix.scala:14)
at com.sebi.count.ScalaSix.main(ScalaSix.scala)
我的代码是:-
package com.sebi.count
import org.apache.spark.sql.SparkSession
object ScalaSix {
def main(args: Array[String]): Unit = {
System.setProperty("hadoop.home.dir", "C:/winutils")
val warehouseLocation = "spark-warehouse"
val spark = SparkSession.builder().appName("Spark Hive Example")
.config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
val df = spark.sql("CREATE TABLE IF NOT EXITS NAME(Id Int, Name String, Age Int,Salary Int)")
val df1 = spark.sql("LOAD DATA LOCAL INPATH 'C:/Users/rittde/IdeaProjects/untitled/src/main/resources' " +
"into table Name")
val df2 = spark.sql("select * from Name")
df2.show()
}
我的sbt依赖项是:-
name := "untitled"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0" libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.0" %"provided"
能否请您建议如何解决该错误。预先感谢。
答案 0 :(得分:-2)
我有同样的错误。我通过更改解决了
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.3" % "provided"
到
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.3"
就我而言。
这是一个说明sbt % "provide"
配置的链接。
SBT assembly jar exclusion