Question

我已解析数据并在RDD后生成：

x [RDD] = (458817,(CompactBuffer(20),CompactBuffer((837063182,0,1433142639864), (676690466,0,1433175090184), (4642913327036075112,1,1433177284025), (464291332,1,1433182403135), (4642913327036075112,0,1433185531150), 
(464291332,0,1433186067803), (4642913327036075112,1,1433186266561), (851805971,0,1433190829047), 
(6376558263039679112,1,1433203286945), (837063182,0,1433226615856), (8403476884799939112,0,1433287740066), 
(764990231,0,1433289484047), (4642913327036075112,0,1433351165901), (464291332,1,1433351892238), 
(4642913327036075112,0,1433374808826), (584492430,1,1433436093253))))

这里我只显示RDD中的记录，我的目标是得到以下RDD：我附加第一个元素的位置。

(458817,837063182,0,1433142639864) 
(458817,676690466,0,1433175090184) 
(458817,464291332,1,1433177284025) 
(458817,464291332,1,1433182403135) 
(458817,464291332,0,1433185531150) 
(458817,464291332,0,1433186067803) 
(458817,464291332,1,1433186266561) 
(458817,851805971,0,1433190829047) 
(458817,637655826,1,1433203286945) 
(458817,837063182,0,1433226615856)

通过执行flatMap我松开了第一个元素并且无法访问它：

val r = x.map(l => l._2).flatMap(x => x._2).map(x => (x._1, x._2, x._3, x._4))

Answer 1

这可能会给你想要的结果：

val r = for {
  el <- Seq(x._1)
  (el1, el2, el3) <- x._2._2
} yield (el, el1, el2, el3)

将第一个元素提升为Sequence以在for表达式中使用它。拉出第二个CompactBuffer并产生想要的元组。

Answer 2

这给了我想要的确切结构。

val s = r.map(x => (x._2._2).map(y => (x._1, y._1, y._2.toInt, y._3, y._4))).flatMap(k => k)

scala flatmap以获得正确的数据结构

2 个答案: