Question

我是scala的新手，只是了解如何通过Map调用进行转换。 foo函数没有被调用。我错过了什么？

import org.apache.spark.rdd.RDD
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

object Expt {

def main(args: Array[String]): Unit = {
        var conf = new SparkConf().setAppName("Test").setMaster("local[*]")
                val sc = new SparkContext(conf)
                val a1 = sc.parallelize(List((1,"one"),(2,"two"),(3,"three"))
                val a3 = a1.map(x => Expt.foo(x))
}

def foo(x: (Int,String)) : (Int,String) = {
        println(x)
        x
}
}

Answer 1

您不执行任何操作，因此永远不会评估地图。地图中的println通常也没有明显的效果。

Answer 2

使用collect或foreach触发计算（执行某些操作）

scala> val a1 = sc.parallelize(List((1,"one"),(2,"two"),(3,"three")))
a1: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[6] at parallelize at <console>:24

scala> val a3 = a1.map(x => foo(x))
a3: org.apache.spark.rdd.RDD[(Int, String)] = MapPartitionsRDD[7] at map at <console>:28

scala> a3.collect
(1,one)
(2,two)
(3,three)
res3: Array[(Int, String)] = Array((1,one), (2,two), (3,three))

scala> a3.foreach(_ => ())
(1,one)
(3,three)
(2,two)

没有从lambda函数调用scala函数

2 个答案: