spark.read()错误,我无法在JAVA中使用Sparksession

时间:2019-06-21 19:19:05

标签: apache-spark

我是Apache Spark的新手,从本地PC读取csv数据时遇到问题。我使用SparkSession读取数据,但收到一些错误消息

我使用此代码

Logger.getLogger("org.apache").setLevel(Level.WARN);
System.setProperty("hadoop.home.dir", "c:/winutils");

SparkSession spark = SparkSession
        .builder()
        .appName("Test")
        .master("local[*]")
        .config("spark.sql.warehouse.dir","file:///c:/tmp/")
        .getOrCreate();

Dataset<Row> ds = spark.read().option("header",false)
                       .csv("src/main/resources/students.csv");
long number=ds.count();
System.out.print("number of 0 " + number );

spark.close();

这是错误消息

  

线程“主”中的异常java.lang.NoSuchMethodError:   scala.Predef $ .refArrayOps([Ljava / lang / Object;)Lscala / collection / mutable / ArrayOps;     在   org.apache.spark.sql.SparkSession $ .org $ apache $ spark $ sql $ SparkSession $$ instantiateSessionState(SparkSession.scala:1103)     在   org.apache.spark.sql.SparkSession $$ anonfun $ sessionState $ 2.apply(SparkSession.scala:145)     在   org.apache.spark.sql.SparkSession $$ anonfun $ sessionState $ 2.apply(SparkSession.scala:144)     在scala.Option.getOrElse(Option.scala:138)在   org.apache.spark.sql.SparkSession.sessionState $ lzycompute(SparkSession.scala:144)     在   org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:141)     在   org.apache.spark.sql.DataFrameReader。(DataFrameReader.scala:789)     在org.apache.spark.sql.SparkSession.read(SparkSession.scala:655)

0 个答案:

没有答案