Question

我对scala和spark很新。试图修复我的spark / scala开发设置。我对版本和丢失的罐子感到困惑。我在stackoverflow上搜索，但仍然坚持这个问题。可能缺少或配置错误。

运行命令：

me@Mycomputer:~/spark-2.1.0$ bin/spark-submit --class ETLApp /home/me/src/etl/target/scala-2.10/etl-assembly-0.1.0.jar

输出：

...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
...
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging

build.sbt：

name := "etl"

version := "0.1.0"
scalaVersion := "2.10.5"
javacOptions ++= Seq("-source", "1.8", "-target", "1.8")
mainClass := Some("ETLApp")

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % "1.5.2";
libraryDependencies += "com.datastax.spark"  %% "spark-cassandra-connector" % "1.5.0-M2";
libraryDependencies += "org.apache.curator" % "curator-recipes" % "2.6.0"
libraryDependencies += "org.apache.curator" % "curator-test" % "2.6.0"
libraryDependencies += "args4j" % "args4j" % "2.32"

java -version java版“1.8.0_101”

scala -version 2.10.5

火花版 2.1.0

欢迎任何提示。感谢

Answer 1

在这种情况下，你的jar必须在提交给spark时带上所有dependend类。

在maven中，这可以通过程序集插件和jar-with-dependencies描述符实现。快速谷歌发现这个：https://github.com/sbt/sbt-assembly

Answer 2

您可以按如下方式更改build.sbt：

name := "etl"

version := "0.1.0"

scalaVersion := "2.10.5"

scalacOptions ++= Seq("-deprecation",
  "-feature",
  "-Xfuture",
  "-encoding",
  "UTF-8",
  "-unchecked",
  "-language:postfixOps")

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.2" % Provided,
"org.apache.spark" %% "spark-sql" % "1.5.2" % Provided,
"org.apache.spark" %% "spark-streaming" % "1.5.2" %  Provided,
"org.apache.spark" %% "spark-streaming-kafka" % "1.5.2" % Provided,
"com.datastax.spark"  %% "spark-cassandra-connector" % "1.5.0-M2",
"org.apache.curator" % "curator-recipes" % "2.6.0",
"org.apache.curator" % "curator-test" % "2.6.0",
"args4j" % "args4j" % "2.32")

mainClass in assembly := Some("your.package.name.ETLApp")

assemblyJarName in assembly := s"${name.value}-${version.value}.jar"

assemblyMergeStrategy in assembly := {
  case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
  case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
  case "reference.conf" => MergeStrategy.concat
  case x: String if x.contains("UnusedStubClass.class") => MergeStrategy.first
  case _ => MergeStrategy.first
}

将sbt-assembly插件添加到Project的Root目录中项目目录下的plugins.sbt文件中。在项目根目录中的终端（Linux）或CMD（Windows）中运行sbt assembly将为您下载所有依赖项并创建一个U

Scala / Spark：引起：java.lang.ClassNotFoundException：org.apache.spark.Logging

2 个答案: