Question

I am running this spark command to run spark Scala program successfully using Hortonworks vm. But once the job is completed it is not exiting from spark-submit command until I hit ctrl+C. Why?

spark-submit --class SimpleApp --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory12m --executor-cores 1 target/scala-2.10/application_2.10-1.0.jar /user/root/decks/largedeck.txt

Here is the code, I am running.

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "YOUR_SPARK_HOME/README.md" // Should be some file on your system
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf) 
    val cards = sc.textFile(args(0)).flatMap(_.split(" "))
    val cardCount = cards.count()
    println(cardCount)
  }
}

Answer 1

你必须在上下文中调用stop（）才能彻底退出程序。

Answer 2

将文件写入S3时遇到了同样的问题。我使用spark 2.0版本，即使在添加了stop（）后如果它不适合你。尝试以下设置

在Spark 2.0中，您可以使用

val spark = SparkSession.builder().master("local[*]").appName("App_name").getOrCreate()
spark.conf.set("spark.hadoop.mapred.output.committer.class","com.appsflyer.spark.DirectOutputCommitter")
spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

spark-submit is not exiting until I hit ctrl+C

2 个答案: