Question

我试图找到一种方法，在给定RDD [Key，Iterable [Value]]类的Iterable [Value]上有效地应用滤波器（多个）。

原因是我想过滤RDD并最终找到与过滤器匹配的键

RDD示例

 000473643-02,CompactBuffer((glucose,80.0), (glucose,80.0), (glucose2,80.0),   (fasting blood glucose,80.0), (glucose,80.0), (glucose,80.0), (glucose,80.0), (glucose,80.0)))
 (713003448-01,CompactBuffer((glucose,80.0), (glucose,80.0)))
 (000023838-01,CompactBuffer((glucose,80.0), (glucose,80.0)))
 (000772974-01,CompactBuffer((glucose,80.0), (glucose,80.0), (glucose,80.0)))
 (380670000-01,CompactBuffer((glucose,80.0), (glucose,80.0)))

所以在这种情况下，我只需要在以下情况为真时输出Key：

    glucose value is >= 80 or fasting blood glucose >= 80

Answer 1

我会用这样的东西：

case class ExceedsCondition(threshold:Double) {
  def violates(value:Double) = value >= threshold
}

val conditionsBroadcast = sc.broadcast(
  Map("glucose" -> ExceedsCondition(80.0), 
      "fasting-glucose" -> ExceedsCondition(81.0))
  )

val rdd = sc.parallelize(List("key1" -> List(("glucose" -> 83.0))))

val result = rdd.filter { case (_, xs) =>
            val conditions = conditionsBroadcast.value
            xs.exists { case (key, value) =>
                conditions.get(key).exists(_.violates(value))
            }
}

result.take(10)

在使用RDD的Scala中，如果在RDD中如何在Iterable中应用函数[（k，Iterable [v]）

1 个答案: