Question

当我运行一个正常的wordcount程序（带有下面的代码）而不包含任何Dataframe时，我可以使用spark-submit运行应用程序。

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
<style type="text/css">
    div:empty { display:none !important;}
</style>
<div class="container">
  <div class="row">
    <div class="col-md-4 col-sm-12"></div> <!--Ensure no white space if no any inside DOM -->
    <div class="col-md-4 col-sm-12">
      <h3>Column 2</h3>
      <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit...</p>
      <p>Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris...</p>
    </div>
    <div class="col-md-4 col-sm-12">
      <h3>Column 3</h3>        
      <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit...</p>
      <p>Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris...</p>
    </div>
  </div>
</div>

但是当我运行以下代码时

object wordCount {   
  def main(args: Array[String]): Unit = {   
 val logFile= "path/thread.txt"    
 val sparkConf = new SparkConf().setAppName("Spark Word Count")    
 val sc = new SparkContext(sparkConf)     
 val file = sc.textFile(logFile)    
 val counts = file.flatMap(_.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)    

 counts.saveAsTextFile("path/output1234")
 sc.stop()    
  }    
}

我收到以下错误：

    import scala.reflect.runtime.universe

    import org.apache.spark.SparkConf
    import org.apache.spark.SparkContext
    import org.apache.spark.rdd.RDD.rddToPairRDDFunctions
object wordCount {    
          def main(args: Array[String]): Unit = {    
           val logFile = "path/thread.txt"    
         val sparkConf = new SparkConf().setAppName("Spark Word Count")    
         val sc = new SparkContext(sparkConf)    
         val file = sc.textFile(logFile)    
         val counts = file.flatMap(_.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)    
        case class count1(key:String,value:Int)    
         val sqlContext =  new org.apache.spark.sql.SQLContext(sc)    
        import sqlContext.implicits._;    
          counts.toDF.registerTempTable("count1")    
           val counts1 = sqlContext.sql("select * from count1")     
         counts.saveAsTextFile("path/output1234")    
         sc.stop()    
          }    
        }

我不确定我错过了什么。

我使用的Pom.xml如下，

Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
 at com.cadillac.spark.sparkjob.wordCount$.main(wordCount.scala:18)

请建议任何更改。

我的群集是

spark-version 2.1.0-mapr-1703 Scala版本2.11.8

提前致谢

Answer 1

如果你转到this documentation，则错误原因定义为

这意味着您的代码中使用的库中存在混合的Scala版本。 Scala 2.10和2.11之间的集合API不同，如果尝试在Scala 2.11运行时加载Scala 2.10库，则会出现最常见的错误。要解决此问题，请确保该名称具有正确的Scala版本后缀以匹配您的Scala版本。

从

更改依赖项

 <dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.10</artifactId>
  <version>1.6.1</version>
 </dependency>
 <dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.10</artifactId>
  <version>1.6.1</version>
</dependency>

到

 <dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.11</artifactId>
  <version>2.1.0</version>
 </dependency>
 <dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.11</artifactId>
  <version>2.1.0</version>
</dependency>

再添加一个依赖

<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.11.8</version>
</dependency>

我猜错误应该消失

数据帧scala.reflect.api.JavaUniverse.runtimeMirror（Ljava / lang / ClassLoader;）错误

1 个答案: