Question

这是我的build.sbt文件的样子：

name := "ProducerExample"

version := "0.1"

scalaVersion := "2.11.12"

run in Compile := { Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)) }

fork in run := true
javaOptions in run ++= Seq(
    "-Dlog4j.debug=true",
    "-Dlog4j.configuration=log4j.properties")

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.4.0",
    "org.apache.spark" %% "spark-streaming-kafka" % "1.6.2",
    "org.apache.kafka" %% "kafka" % "2.2.1"
)

我正在编写一个使用上述库的scala程序。当我在IntelliJ中运行该程序时，它可以工作。现在，我做sbt package，然后找到我创建的jar文件。现在，当我尝试使用以下命令通过spark-submit运行同一程序时：

spark-submit --class ProducerExample /Users/sparker0i/ProducerExample/target/scala-2.11/producerexample_2.11-0.1.jar

我遇到以下错误：

19/06/07 13:07:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/errors/TimeoutException
    at ProducerExample$.main(ProducerExample.scala:16)
    at ProducerExample.main(ProducerExample.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.errors.TimeoutException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 14 more
19/06/07 13:07:52 INFO ShutdownHookManager: Shutdown hook called
19/06/07 13:07:52 INFO ShutdownHookManager: Deleting directory /private/var/folders/7y/xbn9t08j1lbb5fjq03x13hzr0000gn/T/spark-5b3307e0-47f7-4d6a-b5e7-fef0a7f52881

即使我尝试包括以下jar文件：

spark-submit --class ProducerExample /Users/sparker0i/ProducerExample/target/scala-2.11/producerexample_2.11-0.1.jar --jars Downloads/spark-streaming-kafka_2.11-1.6.3.jar --jars Downloads/kafka_2.11-2.2.1.jar

它仍然会引发相同的错误。

编辑：除了执行sbt assembly以外，还有其他方法吗？我可以上传到hdfs的jar文件大小受到限制，因此sbt assembly对我来说不是一个选择。

线程“主”中的异常java.lang.NoClassDefFoundError：org / apache / kafka / common / errors / TimeoutException

0 个答案: