Question

我正在尝试运行Apache Spark示例提供的示例JavaStatefulNetworkWordCount算法，但是当我尝试使用spark submit运行程序时遇到问题，我得到以下异常：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/StateSpec
    at JavaStatefulNetworkWordCount.main(JavaStatefulNetworkWordCount.java:109)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.StateSpec
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

我导入了StateSpec个类，代码与此处提供的代码相同：
https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java

我很感激任何帮助，以了解为什么会出现这个问题以及如何解决它。

Answer 1

您也应该添加实际的提交命令。

我非常怀疑你在提交spark jar时要么缺少--class参数，要么你没有在其中使用正确的合格类名。

Answer 2

我尝试使用以下命令：

spark-submit --class JavaStatefulNetworkWordCount --master local [2] target / SparkStreaming.jar localhost 2222

NoClassDefFound异常Spark Streaming

2 个答案: