Question

我已经检查了其他主题，但我找不到一个好的解决方案，我正在使用intelliJ Idea，我创建了一个SBT项目。

这是我的代码：

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

object mdr {
  def main(args: Array[String]) {
    val logFile = "C:/Users/me/README.md"
    val jarfile = "file:///C:/Users/me/IdeaProjects/untitled/out/artifacts/untitled_jar/untitled.jar"
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]").setJars(Array(jarfile))
    conf.set("spark.eventLog.enabled", "true")
    conf.set("spark.eventLog.dir", "file:///C:/Users/me/spark/logs")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
      val numAs = logData.filter(line => line.contains("a")).count()
      val numBs = logData.filter(line => line.contains("b")).count()
      println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }    
}

这是我的命令行：

spark-submit --driver-class-path C:\Users\me\IdeaProjects\untitled\out\artifacts\untitled_jar\untitled.jar --class mdr  --master local[*]  C:\Users\me\IdeaProjects\untitled\out\artifacts\untitled_jar\untitled.jar

这是我的错误：

java.lang.ClassNotFoundException: mdr
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Unknown Source)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:700)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

感谢Rick Moritz，我发现IntelliJ并没有像预期的那样制作工件，这个主题的最佳答案解决了这个问题：How do I create a runnable JAR in IntelliJ as I would in Eclipse

使用spark-submit提交jar时的ClassNotFoundException

1 个答案: