我有(密钥,值)对的RDD。我需要根据每个键的频率来获取前k个值。
我知道最好的方法是使用combineByKey。
目前这是我的combineByKey组合器的样子
object TopKCount {
//TopK Count combiners
val k: Int = 10
def createCombiner(value: String): Map[String, Long] = {
Map(value -> 1L)
}
def mergeValue(combined: Map[String, Long], value: String): Map[String, Long] = {
combined ++ Map(value -> (combined.getOrElse(value, 0L) + 1L))
}
def mergeCombiners(combined1: Map[String, Long], combined2: Map[String, Long]): Map[String, Long] = {
val top10Keys1 = combined1.toList.sortBy(_._2).takeRight(k).toMap.keys
val top10Keys2 = combined2.toList.sortBy(_._2).takeRight(k).toMap.keys
(top10Keys1 ++ top10Keys2).map(key => (key, combined1.getOrElse(key, 0L) + combined2.getOrElse(key, 0L)))
.toList.sortBy(_._2).takeRight(k).toMap
}
}
我使用如下:
// input is RDD[(String, String)]
val topKValueCount: RDD[(String, Map[String, Long])] = input.combineByKey(
TopKCount.createCombiner,
TopKCount.mergeValue,
TopKCount.mergeCombiners
)
对当前代码的一个优化是在mergeCombiners期间使用 min-queue 。
我更关心网络I / O.有可能一旦我在分区中进行合并,我只会将此分区中的topK条目发送给驱动程序,而不是发送整个Map,这是我在当前情况下所做的。
非常感谢任何反馈。
答案 0 :(得分:0)
为什么不使用Spark的RDD GroupByKey功能或GroupBy?如果您正在使用大型RDD,那么使用Spark功能几乎总是更快吗?
//assuming input is RDD[(String, String)]
val groupinput = input.groupBy(_._2).map(x=>(x._1,x._2.map(y=>y._2).groupBy(identity).map(z=>(z._1,z._2.size)).toList.sortBy(-_._2)))
这个紧凑的1行应该做你想要的。该行首先按键对RDD进行分组,输出RDD(键,Map(键,值))。现在,第二个GroupBy对Mapping的值进行分组,并在新Map中输出这些值的外观频率。
最后,我将地图转换为List(使用数组或您认为合适的任何数据)并对计数(或频率)进行排序。所以你有一个RDD
RDD[(key, List[(value, frequency)])]
现在你可以使用List上的take(k)来获得k个最常见的值。
答案 1 :(得分:0)
我能够如下令人满意地解决这个问题。诀窍是将问题分成两部分,在第一部分中将键及其值组合在一起,得到相同k,v出现的次数,然后将其与新的topk组合器一起使用以获取topk发生值。
case class TopKCount(topK: Int = 10) {
//sort and trim a traversable (String, Long) tuple by _2 value of the tuple
def topNs(xs: TraversableOnce[(String, Long)], n: Int) = {
var ss = List[(String, Long)]()
var min = Long.MaxValue
var len = 0
xs foreach { e =>
if (len < n || e._2 > min) {
ss = (e :: ss).sortBy((f) => f._2)
min = ss.head._2
len += 1
}
if (len > n) {
ss = ss.tail
min = ss.head._2
len -= 1
}
}
ss
}
//seed a list for each key to hold your top N's with your first record
def createCombiner(value: (String, Long)): Seq[(String, Long)] = Seq(value)
//add the incoming value to the accumulating top N list for the key
def mergeValue(combined: Seq[(String, Long)], value: (String, Long)): Seq[(String, Long)] =
topNs(combined ++ Seq((value._1, value._2)), topK)
//merge top N lists returned from each partition into a new combined top N list
def mergeCombiners(combined1: Seq[(String, Long)], combined2: Seq[(String, Long)]): Seq[(String, Long)] =
topNs(combined1 ++ combined2, topK)
}
object FieldValueCount {
//Field Value Count combiners
def createCombiner(value: String): (Double, Long) =
if (Try(value.toDouble).isSuccess) (value.toDouble, 1L)
else (0.0, 1L)
def mergeValue(combined: (Double, Long), value: String): (Double, Long) =
if (Try(value.toDouble).isSuccess) (combined._1 + value.toDouble, combined._2 + 1L)
else (combined._1, combined._2 + 1L)
def mergeCombiners(combined1: (Double, Long), combined2: (Double, Long)): (Double, Long) =
(combined1._1 + combined2._1, combined1._2 + combined2._2)
}
// Example usage. Here input is the RDD[(String, String)]
val topKCount = TopKCount(10)
input.cache()
// combine the k,v from the original input to convert it into (k, (v, count))
val combined: RDD[(String, (String, Long))] = input.map(v => (v._1 + "|" + v._2, 1L))
.reduceByKey(_ + _).map(k => (k._1.split("\\|", -1).head, (k._1.split("\\|", -1).drop(1).head, k._2)))
val topKValueCount: RDD[(String, Seq[(String, Long)])] = combined.combineByKey(
topKCount.createCombiner,
topKCount.mergeValue,
topKCount.mergeCombiners
)
TopKCount
已转换为案例类,以便我们可以根据需要更改k
的值。如果k
不需要变量,则可以将其作为对象。