What is the entry point of Spark container in YARN cluster mode?

Question

What is the main entry point of a Spark executor when a Spark job is being run in Yarn cluster mode (for Spark 1.2.0+)?

What I am looking for is the Scala class name for the entry point of an Executor (which will be the process executing one of the tasks on a slave machine).

Answer 1

spark-submit --class [FULLY QUALIFIED CLASS NAME]
             --master yarn-cluster
             [JAR_TO_USE]

因此，鉴于上述情况，要使用的类是指定的类，它从给定的jar加载，并在该类中搜索static main方法。

来自SparkSubmit.scala：

val mainMethod = mainClass.getMethod("main", new Array[String](0).getClass)

Answer 2

我认为您所询问的是org.apache.spark.executor.Executor或org.apache.spark.executor.Executor$TaskRunner。 TaskRunner最终run a task。

无论是部署模式（client vs cluster）还是使用的集群管理器，即Hadoop YARN或Spark Standalone或Apache Mesos。