Question

我是Scala和Spark的新手。我的Scala代码是：

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf


object Log {

  case class Record(timestamp: String, mtype: String, message: String)

  def main(args: Array[String]){

     val conf = new SparkConf()
    .setAppName("LogProcessing").setMaster("local[2]").set("spark.executor.memory","1g");

    val sc = new SparkContext(conf)
    val sqlContext= new org.apache.spark.sql.SQLContext(sc)
    import sqlContext.implicits._

    val logFile=sc.textFile(PATH_FILE)

    val logFile1=logFile.map(x=> x.split(" - ")).map{
    case Array(timestamp, mtype, message) => Record(timestamp, mtype, message)
    }
    val logFile2=logFile1.toDF()
    logFile2.registerTempTable("Log")

    val messagefromlog=sqlContext.sql("select * from Log")
    messagefromlog.show()
  }
}

我的Pom.xml有以下依赖项：

     <dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>1.3.0</version>
    </dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>1.3.0</version>
</dependency>

  </dependencies>

我已经尝试了其他帖子中提到的所有选项，但没有一个解决了错误。我得到的详细错误是：

Exception in thread "main" scala.ScalaReflectionException: class org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with primordial classloader with boot classpath [Q:\104500.000_ECLIPSE_IDE_FOR_JAVA_EE_DEVELOPERS_450\VFS\USERPROFILE\Eclipse\Eclipse\plugins\org.scala-lang.scala-library_2.11.8.v20160304-115712-1706a37eb8.jar;Q:\104500.000_ECLIPSE_IDE_FOR_JAVA_EE_DEVELOPERS_450\VFS\USERPROFILE\Eclipse\Eclipse\plugins\org.scala-lang.scala-reflect_2.11.8.v20160304-115712-1706a37eb8.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\lib\resources.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\lib\rt.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\lib\sunrsasign.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\lib\jsse.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\lib\jce.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\lib\charsets.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\lib\jfr.jar;
C:\Program Files\Java\jdk1.8.0_40\jre\classes] not found.

JavaMirror中的类org.apache.spark.sql.catalyst.ScalaReflection，其中包含未找到引导类路径的原始类加载器

0 个答案: