Question

我们如何replace spark scala shell中的元素？

例如： val t= sc.parallelize(Seq(("100",List("2","-4","NA","6","8","2"))))

我想用0替换NA

Answer 1

您可以尝试将NA替换为0，但会为您提供新的RDD。

scala> val t= sc.parallelize(Seq(("100",List("2","-4","NA","6","8","2"))))
t: org.apache.spark.rdd.RDD[(String, List[String])] = ParallelCollectionRDD[0] at parallelize at <console>:21
scala> val newRDD = t.map( x => (x._1,x._2.map{case "NA" => 0; case x => x }))
newRDD: org.apache.spark.rdd.RDD[(String, List[Any])] = MapPartitionsRDD[3] at map at <console>:23

scala> newRDD.collect
res5: Array[(String, List[Any])] = Array((100,List(2, -4, 0, 6, 8, 2)))

Answer 2

当你对序列进行并行化时，spark会创建一个提供值的RDD。该RDD存储在spark中的集群中。 RDD的本质是不可变的，另一种方法可能是您可以从RDD中过滤掉“NA”值将它们映射到Int并将每个元素乘以零。并将过滤后的RDD与RDD结合，包括非“NA”元素。

示例代码

val t= sc.parallelize(Seq(("100",List("2","-4","NA","6","8","2"))))
val a = t.map(i => i._2).filter(i => i.contains("NA"))
val b = t.map(i => i._2).filter(i => !i.contains("NA")).map(i => (i*0))
val d = a.union(b)

替换spark scala中的元素

2 个答案: