Question

我每天运行火花测试几次到很多次，但是在启动一个简单的独立应用程序时遇到了一个新问题。我已经在spark 2.3.0和2.4.0上尝试了相同的结果。

  def main(args: Array[String]): Unit = {
    val spark =   SparkSession.builder.appName("bestRoutes").master("local").getOrCreate
    // do stuff with the session..
  }

这会导致错误/堆栈跟踪显示在这里：

19/03/09 11:02:24 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.4:54406 with 2004.6 MB RAM, BlockManagerId(driver, 192.168.0.4, 54406, None)
19/03/09 11:02:24 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.4, 54406, None)
19/03/09 11:02:24 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.4, 54406, None)
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/StaticSQLConf$
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
    at task1.AirData$$anonfun$2.apply(AirData.scala:52)
    at task1.AirData$$anonfun$2.apply(AirData.scala:52)
    at scala.Option.getOrElse(Option.scala:121)
    at task1.AirData$.main(AirData.scala:51)
    at task1.AirData.main(AirData.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.internal.StaticSQLConf$
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 6 more
19/03/09 11:02:24 INFO SparkContext: Invoking stop() from shutdown hook
19/03/09 11:02:24 INFO SparkUI: Stopped Spark web UI at http://192.168.0.4:4040

我使用的是经过验证的真实maven pom.xml-此处显示了相关的spark部分：

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>

然后是必需的阴影-这是spark项目的标准部分

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <createDependencyReducedPom>false</createDependencyReducedPom>
                        <relocations>
                            <relocation>
                                <pattern>com.google.common</pattern>
                                <shadedPattern>shaded.com.google.common</shadedPattern>
                            </relocation>
                            <relocation>
                                <pattern>org.glassfish.hk2</pattern>
                            </relocation>

                        </relocations>
                        <transformers>
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <manifestEntries>
                                    <Main-Class>a2a.TimeSeries</Main-Class>
                                    <Build-Number>1.0</Build-Number>
                                </manifestEntries>
                            </transformer>
                        </transformers>
                        <artifactSet>
                            <excludes>
                                <exclude>org.apache.spark:*</exclude>
                                <exclude>org.apache.hadoop:*</exclude>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <artifact>*.*</artifact>
                                <excludes>
                                    <!--<exclude>org.apache.maven:lib:tests</exclude>-->
                                    <exclude>log4j:log4j:jar:</exclude>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                    <exclude>META-INF/ECLIPSE*</exclude>
                                    <exclude>META-INF/license/*</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <minimizeJar>false</minimizeJar>
                    </configuration>
                </execution>
            </executions>
        </plugin>

Answer 1

StaticSQLConf类是工件spark-catalyst _ $ {scala.binary.version}的一部分，用作提供的Maven范围。因此，spark运行时找不到它。应该可以编译。

ReadP

启动简单的独立Spark应用程序时出现问题：java.lang.NoClassDefFoundError：org / apache / spark / sql / internal / StaticSQLConf $

1 个答案: