使用Spark计算句子之间的相似性

Question

我的问题陈述有以下输入： -

 ID  -> List of Words
(101 -> Array("a1","b2","c4","d2"))
(102 -> Array("a6","b1","c5","d3"))
(103 -> Array("a1","b4","c4","d2"))
(104 -> Array("a2","b2","c3","d2"))
(105 -> Array("a7","b6","c1","d3"))

现在，我想找出这些输入语句之间的相似性。

示例： -

(101 -> Array("a1","b2","c4","d2"))
(103 -> Array("a1","b4","c4","d2"))
(104 -> Array("a2","b2","c3",",d2"))

在示例输出中，语句彼此非常相似。

如何使用Spark实现此目的？我可以使用任何逻辑代码或任何机器学习算法。

由于