Question

我有一个简单的Java Spark应用程序（SimpleApp.jar），我正在Windows环境中使用以下命令尝试运行该应用程序

spark-submit.cmd --class SimpleJar --master local SimpleApp.jar

不断抛出

java.lang.ClassNotFoundException: SimpleApp
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Unknown Source)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我的项目的目录结构是这样的

SimpleApp
- infrastructure (AN ANSIBLE MODULE)
- SimpleApp (A MAVEN MODULE)
- - src
- - - main
- - - - java
- - - - - SimpleApp.java
- - - - resources
- - pom.xml

在POM内，我正在使用shade插件像这样构建我的uber jar

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <!-- Run shade goal on package phase -->
                    <execution>
                        <id>SimpleAppId</id>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <!-- add Main-Class to manifest file -->
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>SimpleApp</mainClass>
                                </transformer>
                            </transformers>
                            <shadedClassifierName>SimpleApp</shadedClassifierName>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

有人可以建议在这里配置错误的内容。这与Windows环境有关吗？我可以毫无问题地将其部署在AWS EMR上。此外，这也可能有用

unzip -p SimpleApp.jar META-INF/MANIFEST.MF

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Built-By: <author>
Created-By: Apache Maven 3.3.9
Build-Jdk: 1.8.0_191
Main-Class: SimpleApp

Answer 1

我的机器上运行的spark和我在应用程序中使用的依赖项之间存在版本不匹配的情况。一旦我更改为最新版本的spark，一切都会顺利进行。

spark-submit抛出java.lang.ClassNotFoundException：SparkPIIDetector

1 个答案: