Question

我正在尝试使用spark来分析facebook网络。当我创建Spark Conf的对象时，它给了我以下错误。线程“main”中的异常java.lang.NoClassDefFoundError：org / apache / spark / SparkConf。

我正在使用scala版本2.11，它应该可以正常工作。在我的intelliJ IDEA中也加载了scala sdk 2.12。代码相同的是： -

import org.apache.spark.SparkConf
import org.apache.spark.graphx.GraphLoader
import org.apache.spark.sql.SparkSession

object Sample {

   def main(args: Array[String]): Unit = {
    print("Hello World")

    val conf = new SparkConf().setMaster("local[2]")
    val spark = SparkSession
      .builder
      .appName("SampleApp").config(conf)
      .getOrCreate()
    val sc = spark.sparkContext

    val graph = GraphLoader.edgeListFile(sc, "facebook_combined.txt")

    val ranks = graph.pageRank(1).vertices

    print(ranks.collect().mkString("\n"))

    spark.stop()

  }
}

我之前说过我的pom.xml文件： -

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>Sample</groupId>
<artifactId>CMPE256</artifactId>
<version>1.0-SNAPSHOT</version>



<properties>
    <spark.version>2.1.1</spark.version>
    <scala.dep.version>2.11</scala.dep.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.dep.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.dep.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-graphx_${scala.dep.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.6.6</version>
    </dependency>
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.14</version>
    </dependency>
</dependencies>



<build>
    <plugins>
        <!-- mixed scala/java compile -->
        <plugin>
            <groupId>org.scala-tools</groupId>
            <artifactId>maven-scala-plugin</artifactId>
            <version>2.15.2</version>
            <executions>
                <execution>
                    <id>compile</id>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                    <phase>compile</phase>
                </execution>
                <execution>
                    <id>test-compile</id>
                    <goals>
                        <goal>testCompile</goal>
                    </goals>
                    <phase>test-compile</phase>
                </execution>
                <execution>
                    <phase>process-resources</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.6.1</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
  </build>
</project>

Answer 1

注释掉<scope>provided</scope>中的pom.xml，这应该可以解决问题。

有关为什么在这里的更多信息：MavenDoc

Answer 2

你可以在 Scala 中这样写。

    package Wordcount
    import org.apache.spark.streaming._
    import org.apache.spark.SparkContext
    import org.apache.spark._

    class FirstApp {
    val conf = new SparkConf().setMaster("local[2]").setAppName("wordcount");
    ...
    }

找不到SparkConf类

2 个答案: