Question

Spark-Scala中以下pySpark代码的等效内容是什么？

rddKeyTwoVal = sc.parallelize([("cat", (0,1)), ("spoon", (2,3))])
rddK2VReorder = rddKeyTwoVal.map(lambda (key, (val1, val2)) : ((key, val1) ,
val2))
rddK2VReorder.collect()
// [(('cat', 0), 1), (('spoon', 2), 3)] -- This is the output.

Answer 1

val rddKeyTwoVal = sc.parallelize(Seq(("cat", (0,1)), ("spoon", (2,3))))
val rddK2VReorder = rddKeyTwoVal.map{case (key, (val1, val2)) => ((key, val1), val2)}
rddK2VReorder.collect

或

val rddKeyTwoVal = sc.parallelize(Seq(("cat", (0,1)), ("spoon", (2,3))))
val rddK2VReorder = rddKeyTwoVal.map(r=> ((r._1, r._2._1),r._2._2))
rddK2VReorder.collect

输出：

 Array(((cat,0),1), ((spoon,2),3))

感谢@Alec建议第一种方法

Answer 2

我找到了自己的答案！发布以帮助社区的其他人。这是我上面发布的代码中最干净的Scala版本。产生完全相同的输出。

val rddKeyTwoVal = sc.parallelize(Array(("cat", (0,1)), ("spoon", (2,3))))
val rddK2VReorder = rddKeyTwoVal.map{case (key, (val1, val2)) => ((key, val1),val2)}

rddK2VReorder.collect()

//Use the following for a cleaner output. 
rddK2VReorder.collect().foreach(println)

输出：

// With collect() menthod.

Array[((String, Int), Int)] = Array(((cat,0),1), ((spoon,2),3))

// If you use the collect().foreach(println)
((cat,0),1)
((spoon,2),3)

使用Spark Scala中的map（）重新排序键值对

2 个答案: