Question

这不是一个重复的问题，我已经尝试了很多方法来完成这项工作，但没有成功

我正在尝试编写一个字数统计应用程序，以便我可以通过提交火花来运行

我正在使用IntelliJ IDEA，spark-2.1.1和scala-2.11.8

我的字数统计代码看起来像这样::

package com.netflix.utilities

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

class SparkWordCount {
  def main(args: Array[String]) {
   // create Spark context with Spark configuration
   println("starting")
   val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

   // get threshold
   val threshold = args(1).toInt

   // read in text file and split each document into words
   val tokenized = sc.textFile(args(0)).flatMap(_.split(" "))

   // count the occurrence of each word
   val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)

   // filter out words with fewer than threshold occurrences
   val filtered = wordCounts.filter(_._2 >= threshold)

   // count characters
   val charCounts = filtered.flatMap(_._1.toCharArray).map((_, 1)).reduceByKey(_ + _)

   System.out.println(charCounts.collect().mkString(", "))
  }
}

我的pom.xml看起来像这样::

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.netflix.utilities</groupId>
<artifactId>SparkWordCount</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>

</dependencies>

<distributionManagement>
    <repository>
        <id>internal.repo</id>
        <name>Internal repo</name>
        <url>file:///Users/sankar.biswas/Noah/</url>
    </repository>
</distributionManagement>


<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

现在我将我的罐子放在了我提交火花的同一文件夹中

这是我正在运行的火花提交::

./ spark-submit --class com.netflix.utilities.SparkWordCount --master本地--deploy-mode客户端SparkWordCount-1.0-SNAPSHOT.jar“ file：////Users/sankar.biswas/Desktop/ hello.txt”

但是我不断收到此错误::

java.lang.ClassNotFoundException：SparkWordCount

我没有得到我所缺少的东西。任何建议将不胜感激！

Answer 1

代替对象SparkWordCount，使用对象SparkWordCount并将--class“ SparkWordCount”更改为--class com.netflix.utilities.SparkWordCount并运行

SparkWordCount问题-java.lang.ClassNotFoundException

1 个答案: