我刚刚提出了在DSE集群4.8上运行Spark(pyspark)的奇怪问题。出于某种原因,当作业运行时,我有几十个错误,如下所述。有人知道有什么办法可以关闭这个错误吗?
我正在使用dse spark-submit
命令。
ERROR 2016-04-04 12:49:21,305 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 747
ERROR 2016-04-04 12:49:21,305 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 748
ERROR 2016-04-04 12:49:21,347 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 749
ERROR 2016-04-04 12:49:21,388 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 750
答案 0 :(得分:0)
When I see this message "Asked to remove non-existent executor" that means the executor crashed for a different reason and so obviously spark can't kill it. Look at the executor logs to find out why it has crashed.
It's also possible you have a networking problem and your driver and executor can't communicate. Have also seen this message in that case.