Question

我很新兴火花，我正试图尝试火花提交。我在spring boot中创建了一个应用程序，使用mvn package创建了一个jar。但是当我尝试将jar提交到spark-submit时，无法找到Main类。但主要的课程存在于罐子里。

 spark-submit --class com.dip.sparkapp.SparkappApplication --master local target/sparkapp-0.0.1-SNAPSHOT.jar

Answer 1

实际上，我们遇到了同样的问题，就在你发布这个问题的同一天。我们的解决方案是使用maven的shade插件来编辑我们的构建。我们发现，在使用spring-boot-maven插件进行打包时，它将我们的类嵌套在BOOT-INF /类中，这些类似于火花不喜欢的类。我会粘贴相关部分，以便您可以在自己的应用程序中试用它 - 祝你好运！

<build>
   <plugins>
      <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <dependencies>
                <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                    <version>${spring-boot-version}</version>
                </dependency>
            </dependencies>
            <configuration>
                <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
                <createDependencyReducedPom>false</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.handlers</resource>
                    </transformer>
                    <transformer
                            implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                        <resource>META-INF/spring.factories</resource>
                    </transformer>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.schemas</resource>
                    </transformer>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>${start-class}</mainClass>
                    </transformer>
                </transformers>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Answer 2

我发现只是从spark-submit提交中跳过类名即可，即--class com.dip.sparkapp.SparkappApplication

Answer 3

这对我有用

<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
    <manifestEntries>
        <Main-Class>packagename.classname</Main-Class>
    </manifestEntries>
</transformer>

Answer 4

如果您使用的是gradle，则适用于影子插件：

import com.github.jengelman.gradle.plugins.shadow.transformers.*

...

plugins {
    id 'com.github.johnrengelman.shadow' version '2.0.4'
}

...

shadowJar {
    zip64 true
    mergeServiceFiles()
    append 'META-INF/spring.handlers'
    append 'META-INF/spring.schemas'
    append 'META-INF/spring.tooling'
    transform(PropertiesFileTransformer) {
        paths = ['META-INF/spring.factories' ]
        mergeStrategy = "append"
    }
}

从这里拍摄：https://github.com/spring-projects/spring-boot/issues/1828#issuecomment-607352468

提交spring boot应用程序jar以spark-submit

4 个答案: