Question

  def map[U: ClassTag](f: T => U): RDD[U] = withScope {
    val cleanF = sc.clean(f)
    new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.map(cleanF))
  }

此代码段来自spark 2.2源代码。我在scala方面不专业，所以只是想知道任何人都可以从程序化角度解释这段代码吗？我不确定地图后的方括号是什么。并且还可以参考https://www.tutorialspoint.com/scala/scala_functions.htm，scala函数应该在'='后面有一个大括号，但为什么在这个代码片段中有一个名为withScope的函数在'='之后？

Answer 1

实际上，在＃34; =＆＃34;之后，scala函数不能有任何括号。例如

def func():Int = 1

所以你可以认为withScope {}是一个返回类型为RDD [U]的函数，map函数用于运行withScope函数。

让我们看一下withScope的源代码：

private[spark] def withScope[U](body: => U): U = RDDOperationScope.withScope[U](sc)(body)

看，这是一个功能。让我们继续：

private[spark] def withScope[T](
    sc: SparkContext,
    allowNesting: Boolean = false)(body: => T): T = {
    val ourMethodName = "withScope"
    val callerMethodName = Thread.currentThread.getStackTrace()
    .dropWhile(_.getMethodName != ourMethodName)
    .find(_.getMethodName != ourMethodName)
    .map(_.getMethodName)
    .getOrElse {
      // Log a warning just in case, but this should almost certainly never happen
      logWarning("No valid method name for this RDD operation scope!")
      "N/A"
    }
  withScope[T](sc, callerMethodName, allowNesting, ignoreParent = false)(body)
}

最后继续withScope：

private[spark] def withScope[T](
    sc: SparkContext,
    name: String,
    allowNesting: Boolean,
    ignoreParent: Boolean)(body: => T): T = {
  // Save the old scope to restore it later
  val scopeKey = SparkContext.RDD_SCOPE_KEY
  val noOverrideKey = SparkContext.RDD_SCOPE_NO_OVERRIDE_KEY
  val oldScopeJson = sc.getLocalProperty(scopeKey)
  val oldScope = Option(oldScopeJson).map(RDDOperationScope.fromJson)
  val oldNoOverride = sc.getLocalProperty(noOverrideKey)
  try {
    if (ignoreParent) {
      // Ignore all parent settings and scopes and start afresh with our own root scope
      sc.setLocalProperty(scopeKey, new RDDOperationScope(name).toJson)
    } else if (sc.getLocalProperty(noOverrideKey) == null) {
      // Otherwise, set the scope only if the higher level caller allows us to do so
      sc.setLocalProperty(scopeKey, new RDDOperationScope(name, oldScope).toJson)
    }
    // Optionally disallow the child body to override our scope
    if (!allowNesting) {
      sc.setLocalProperty(noOverrideKey, "true")
    }
    body
  } finally {
    // Remember to restore any state that was modified before exiting
    sc.setLocalProperty(scopeKey, oldScopeJson)
    sc.setLocalProperty(noOverrideKey, oldNoOverride)
  }
}

最后，它执行body参数，在这种情况下，body等于

{
  val cleanF = sc.clean(f)
  new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.map(cleanF))
}

总之，withScope是一个Closure接受一个函数作为参数，它首先运行一些代码本身并运行参数。

需要一些解释才能理解scala中的map函数

1 个答案: