如何知道Spark应用程序的执行程序IP？

Question

我想在运行时获取执行程序的所有IP，我应该使用Spark中的哪个API？或者在运行时获取IP的任何其他方法？

Answer 1

Apache Spark中有一个类，即ExecutorInfo，它有一个返回Executor主机IP的方法executorHost（）。

Answer 2

您应该使用SparkListener抽象类并拦截两个特定于执行程序的事件 - SparkListenerExecutorAdded和SparkListenerExecutorRemoved。

override def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit = {
  val execId = executorAdded.executorId
  val host = executorAdded.executorInfo.executorHost
  executors += (execId -> host)
  println(s">>> executor id=$execId added on host=$host")
}

override def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit = {
  val execId = executorRemoved.executorId
  val host = executors remove execId getOrElse "Host unknown"
  println(s">>> executor id=$execId removed from host=$host")
}

整个工作项目都在我的Spark Executor Monitor Project。