Question

当我在笔记本的databrick上运行spark作业时，它不会打印最终结果，而spark告诉我它已经跳过了某个阶段。代码如下：


import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.log4j._

/** Count up how many of each word occurs in a book, using regular expressions and sorting the final results */
    // Set the log level to only print errors
    Logger.getLogger("org").setLevel(Level.ERROR)

    // Load each line of my book into an RDD
    val input = sc.textFile("/FileStore/tables/book.txt")

    // Split using a regular expression that extracts words
    val words = input.flatMap(x => x.split("\\W+"))

    // Normalize everything to lowercase
    val lowercaseWords = words.map(x => x.toLowerCase())

    // Count of the occurrences of each word
    val wordCounts = lowercaseWords.map(x => (x, 1)).reduceByKey( (x,y) => x + y )

    // Flip (word, count) tuples to (count, word) and then sort by key (the counts)
    val wordCountsSorted = wordCounts.map( x => (x._2, x._1) ).sortByKey()

    // Print the results, flipping the (count, word) results to word: count as we go.
   for (result <- wordCountsSorted) {
      val count = result._1
      val word = result._2
      println(s"$word: $count")
    }

我得到以下日志：

(2) Spark Jobs
Job 2 View(Stages: 2/2)
Job 3 View(Stages: 2/2, 1 skipped)

Stage 7: 
0/2succeeded / total tasks skipped
Stage 8: 
2/2succeeded / total tasks 
Stage 9: 
2/2succeeded / total tasks 


import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.log4j._
conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@3b172b00
sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@4e94620c
input: org.apache.spark.rdd.RDD[String] = /FileStore/tables/book.txt MapPartitionsRDD[11] at textFile at command-85753:21
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[12] at flatMap at command-85753:24
lowercaseWords: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[13] at map at command-85753:27
wordCounts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[15] at reduceByKey at command-85753:30
wordCountsSorted: org.apache.spark.rdd.RDD[(Int, String)] = ShuffledRDD[19] at sortByKey at command-85753:33

我认为之所以看不到任何打印输出，是因为火花跳过了工作阶段7，但是如何防止这种情况发生？

Answer 1

我解决了。我只需要在wordCountsSorted.collect()之后加上wordCountsSorted。

在databrick笔记本上的多个群集上运行时，Spark作业无法打印

1 个答案: