我认为它与内存有关，因为它适用于较小的数据集。该程序在使用Spark-Mllib的Logistic回归时使用并过早关闭。我在下面运行此命令在HDFS上启动我的spark程序。
export SPARK_CONF_DIR=/home/gs/conf/spark/latest
export SPARK_HOME=/home/gs/spark/latest
$SPARK_HOME/bin/spark-submit --class algoRunner --master yarn --deploy-mode cluster --conf spark.dynamicAllocation.enabled=true \
--executor-memory 8g --queue default --conf spark.hadoop.hive.querylog.location='${java.io.tmpdir}/hivelogs' \
~/spark/Product-Classifier-Pipeline-assembly-1.0.jar
我收到以下错误：
17/08/02 21:53:40 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
17/08/02 21:53:40 INFO SparkContext: Invoking stop() from shutdown hook
17/08/02 21:53:40 INFO SparkUI: Stopped Spark web UI at http://gsrd219n01.red.ygrid.yahoo.com:45546
17/08/02 21:53:40 INFO DAGScheduler: Job 10 failed: treeAggregate at LogisticRegression.scala:1670, took 2.351935 s
17/08/02 21:53:40 INFO DAGScheduler: ShuffleMapStage 19 (treeAggregate at LogisticRegression.scala:1670) failed in 1.947 s due to Stage cancelled because SparkContext was shut down
17/08/02 21:53:40 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo@21bec75d)
17/08/02 21:53:40 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(10,1501710820713,JobFailed(org.apache.spark.SparkException: Job 10 cancelled because SparkContext was shut down))
17/08/02 21:53:40 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job 10 cancelled because SparkContext was shut down
org.apache.spark.SparkException: Job 10 cancelled because SparkContext was shut down
答案 0 :(得分：0)
驱动程序内存不够大。增加它可以防止这些错误。