高效计算集合元素之间的正弦距离

时间:2019-03-22 16:33:03

标签: scala

我有两个收藏。每个集合都由一个包含纬度,经度和纪元的集合组成。

val arr1= Seq(Seq(34.464, -115.341,1486220267.0), Seq(34.473, 
-115.452,1486227821.0), Seq(35.572, -116.945,1486217300.0), 
Seq(37.843, -115.874,1486348520.0),Seq(35.874, -115.014,1486349803.0), 
Seq(34.345, -116,924, 1486342752.0) )

val arr2= Seq(Seq(35.573, -116.945,1486217300.0 ),Seq(34.853, 
-114.983,1486347321.0 ) )

我想确定两个数组在.5英里内且具有相同历元的次数。我有两个功能

def haversineDistance_single(pointA: (Double, Double), pointB: (Double, Double)): Double = {
  val deltaLat = math.toRadians(pointB._1 - pointA._1)
  val deltaLong = math.toRadians(pointB._2 - pointA._2)
  val a = math.pow(math.sin(deltaLat / 2), 2) + math.cos(math.toRadians(pointA._1)) * math.cos(math.toRadians(pointB._1)) * math.pow(math.sin(deltaLong / 2), 2)
  val greatCircleDistance = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))
  3958.761 * greatCircleDistance
    }

def location_time(col_2:Seq[Seq[Double]], col_1:Seq[Seq[Double]]): Int={
  val arr=col_1.map(x=> col_2.filter(y=> (haversineDistance_single((y(0), y(1)), (x(0),x(1)))<=.5) &

    (math.abs(y(2)-x(2))<=0)).flatten).filter(x=> x.length>0)
  arr.length
}


location_time(arr1,arr2) =1

我的实际集合非常大,有没有比location_time函数更有效的方法来计算该集合。

1 个答案:

答案 0 :(得分:2)

我会考虑修改location_time的来源:

def location_time(col_mobile: Seq[Seq[Double]], col_laptop: Seq[Seq[Double]]): Int = {
  val arr = col_laptop.map( x => col_mobile.filter( y =>
      (haversineDistance_single((y(0), y(1)), (x(0), x(1))) <= .5) & (math.abs(y(2) - x(2)) <= 0)
    ).flatten
  ).filter(x => x.length > 0)

  arr.length
}

收件人:

def location_time(col_mobile: Seq[Seq[Double]], col_laptop: Seq[Seq[Double]]): Int = {
  val arr = col_laptop.flatMap( x => col_mobile.filter( y =>
      ((math.abs(y(2) - x(2)) <= 0 && haversineDistance_single((y(0), y(1)), (x(0), x(1))) <= .5))
    )
  )

  arr.length
}

所做的更改:

  1. col_mobile.filter(y => ...)的修订版:

    filter(_ => costlyCond1 & lessCostlyCond2)
    

    收件人:

    filter(_ => lessCostlyCond2 && costlyCond1)
    

    假设haversineDistance_single的运行成本比math.abs高,将&替换为&&(请参见& versus &&之间的区别)并测试math.abs首先可能会提高过滤性能。

  2. 使用map/filter/flatten/filter简化了flatMap,替换为:

    col_laptop.map(x => col_mobile.filter(y => ...).flatten).filter(_.length > 0)
    

    具有:

    col_laptop.flatMap( x => col_mobile.filter( y => ... ))
    

例如,如果您可以访问Apache Spark集群,请考虑将您的集合(如果它们确实很大)转换为RDD,以使用与上述类似的转换进行计算。