Question

如果我们有一个model.approxSimilarityJoin的数据框输出

val results =
  model
   .approxSimilarityJoin(vectorizedDf, vectorizedDf, threshold)
   .filter("distCol != 0")
   .filter("distCol < 0.2")
   .select(col("datasetA.title").alias("idA"), col("datasetB.title").alias("idB"), col("distCol"))

输出上述命令

**idA|idB|distCol**  
A|B|0.125 
B|C|0.125 
A|D|0.125 
D|E|0.125  
F|G|0.125 
X|Y|0.19 
A|M|0.14 
A|N|0.14

我们希望将输出分组并计算类似的项目，即上例中的输出我们有

A, B, C, D, E  
F,G
X,Y

所需的最终输出应该是：

A, 0.125, 5
F, 0.19, 1
A, 0.14, 2

Answer 1

在spark-testing-base

中有一个叫做大约DataFrame相等性检查的东西

  /**
    * Compares if two [[DataFrame]]s are equal, checks that the schemas are the same.
    * When comparing inexact fields uses tol.
    *
    * @param tol max acceptable tolerance, should be less than 1.
    */
  def assertDataFrameApproximateEquals(
    expected: DataFrame, result: DataFrame, tol: Double) {

    assert(expected.schema, result.schema)

    try {
      expected.rdd.cache
      result.rdd.cache
      assert("Length not Equal", expected.rdd.count, result.rdd.count)

      val expectedIndexValue = zipWithIndex(expected.rdd)
      val resultIndexValue = zipWithIndex(result.rdd)

      val unequalRDD = expectedIndexValue.join(resultIndexValue).
        filter{case (idx, (r1, r2)) =>
          !DataFrameSuiteBase.approxEquals(r1, r2, tol)}

      assertEmpty(unequalRDD.take(maxUnequalRowsToShow))
    } finally {
      expected.rdd.unpersist()
      result.rdd.unpersist()
    }
  }

根据您的需要设计算法。 https://github.com/holdenk/spark-testing-base/wiki/DataFrameSuiteBase

Answer 2

我们正试图找到像Natural Join这样的东西。在上面的例子中， A与B有关，B与C有关，即A与C有关。加上我们有 A与D有关，D与E有关，即A与E有关

最后我们应该得出结论，A与B，C，D和E有相同的相似性即A已计入5

为aboutSimilarityJoin分组并计算spark数据帧

2 个答案: