Question

我正在尝试使用SBT读取XML文件，但在编译时我遇到了问题。

build.sbt

name:= "First Spark"
version:= "1.0"
organization := "in.goai"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0"
libraryDependencies += "com.databricks" % "spark-avro_2.10" % "2.0.1"
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.0.2"
resolvers += Resolver.mavenLocal

.scala文件

package in.goai.spark

import scala.xml._
import com.databricks.spark.xml
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkContext, SparkConf}

object SparkMeApp {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("First Spark")
    val sc = new SparkContext(conf)
    val sqlContext = new SQLContext(sc)
    val fileName = args(0)
    val df = sqlContext.read.format("com.databricks.spark.xml").option("rowTag", "book").load("fileName")
    val selectedData = df.select("title", "price")
    val d = selectedData.show
    println(s"$d")

  }
}

当我通过提供＆＃34; sbt package＆＃34;来编译它时它显示下面的错误

[error] /home/hadoop/dev/first/src/main/scala/SparkMeApp.scala:4: object xml is not a member of package com.databricks.spark
[error] import com.databricks.spark.xml
[error]        ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 9 s, completed Sep 22, 2017 4:11:19 PM

我是否需要添加与xml相关的任何其他jar文件？请建议并请提供任何链接，提供有关不同文件格式的jar文件的信息

Answer 1

因为您在build.sbt中使用了Scala 2.11和Spark 2.0，所以将依赖项更改为以下内容：

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0"
libraryDependencies += "com.databricks" %% "spark-avro" % "3.2.0"
libraryDependencies += "com.databricks" %% "spark-xml" % "0.4.1"
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.0.6"

将spark-avro版本更改为3.2.0：https://github.com/databricks/spark-avro#requirements
添加"com.databricks" %% "spark-xml" % "0.4.1"：https://github.com/databricks/spark-xml#scala-211
将scala-xml版本更改为1.0.6，Scala 2.11的当前版本：http://mvnrepository.com/artifact/org.scala-lang.modules/scala-xml_2.11

在您的代码中，删除以下import语句：

import com.databricks.spark.xml

请注意，您的代码实际上并未使用spark-avro或scala-xml库。如果您不打算使用它们，请从build.sbt（以及代码中的import scala.xml._语句中删除这些依赖项。

错误：对象xml不是包com.databricks.spark的成员

1 个答案: