首先，它是你告诉它的任何东西

--num-executors 4

如果使用动态分配，那么这就是它决定你的方式

根据这份文件（http://jerryshao.me/architecture/2015/08/22/spark-dynamic-allocation-investigation/），

Spark如何通过待处理和正在运行的任务计算其所需的最大执行程序数：

 private def maxNumExecutorsNeeded(): Int = {
    val numRunningOrPendingTasks = listener.totalPendingTasks + listener.totalRunningTasks
    (numRunningOrPendingTasks + tasksPerExecutor - 1) / tasksPerExecutor
 }

如果当前执行人数超过预期数量：

 // The target number exceeds the number we actually need, so stop adding new
 // executors and inform the cluster manager to cancel the extra pending requests
 val oldNumExecutorsTarget = numExecutorsTarget
 numExecutorsTarget = math.max(maxNeeded, minNumExecutors)
 numExecutorsToAdd = 1

 // If the new target has not changed, avoid sending a message to the cluster manager
 if (numExecutorsTarget < oldNumExecutorsTarget) {
   client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
   logDebug(s"Lowering target number of executors to $numExecutorsTarget (previously " +
     s"$oldNumExecutorsTarget) because not all requested executors are actually needed")
 }
 numExecutorsTarget - oldNumExecutorsTarget

如果当前执行程序编号超过所需编号，Spark将通知集群管理器取消挂起的请求，因为它们是不需要的。对于那些已经分配了执行者的人，他们将通过超时机制逐渐降低到合理的数量。

如果当前执行人数不能满足所需的数字：

 val oldNumExecutorsTarget = numExecutorsTarget

 // There's no point in wasting time ramping up to the number of executors we already have, so
 // make sure our target is at least as much as our current allocation:
 numExecutorsTarget = math.max(numExecutorsTarget, executorIds.size)

 // Boost our target with the number to add for this round:
 numExecutorsTarget += numExecutorsToAdd

 // Ensure that our target doesn't exceed what we need at the present moment:
 numExecutorsTarget = math.min(numExecutorsTarget, maxNumExecutorsNeeded)

 // Ensure that our target fits within configured bounds:
 numExecutorsTarget = math.max(math.min(numExecutorsTarget, maxNumExecutors), minNumExecutors)
 val delta = numExecutorsTarget - oldNumExecutorsTarget

 // If our target has not changed, do not send a message
 // to the cluster manager and reset our exponential growth
 if (delta == 0) {
   numExecutorsToAdd = 1
   return 0
 }
 val addRequestAcknowledged = testing ||
   client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
 if (addRequestAcknowledged) {
   val executorsString = "executor" + { if (delta > 1) "s" else "" }
   logInfo(s"Requesting $delta new $executorsString because tasks are backlogged" +
     s" (new desired total will be $numExecutorsTarget)")
   numExecutorsToAdd = if (delta == numExecutorsToAdd) {
     numExecutorsToAdd * 2
   } else {
     1
   }
   delta
 } else {
   logWarning(
     s"Unable to reach the cluster manager to request $numExecutorsTarget total executors!")
   0
 }

Answer 2

两个可能的答案：

如果您在调用spark-submit时指定执行者数量，则应获得您要求的金额--num-executors X
如果您没有指定，则默认情况下Spark应使用动态分配，如果需要，将启动更多执行程序。在这种情况下，您可以配置行为，例如最大执行者数量，请参阅http://spark.apache.org/docs/latest/configuration.html#dynamic-allocation

每个工作节点的执行程序数量将取决于可用资源。

在spark中为每个工作节点运行多少个执行程序进程？

2 个答案:

首先，它是你告诉它的任何东西

如果使用动态分配，那么这就是它决定你的方式