Question

我有2个Spark数据帧

library(SparkR); library(magrittr)

df1 <- createDataFrame(data.frame(var1 = c("rat", "cat", "bat")))
df2 <- createDataFrame(data.frame(var2 = c("cat3", "bat1", "dog", "toy")))

我需要使用 SparkR的levenshtein 函数模糊匹配来自不同Spark DataFrames df1和df2的不同长度的var1和var2，以便获得所需的输出。

desired_df <- createDataFrame(data.frame(var2 = c("cat3", "bat1", "dog", "toy"),
                                  var3 = c("cat", "bat", NA_character_, NA_character_)))

我从以下代码开始：

df3 <- df2 %>% SparkR::mutate(dist = levenshtein(df2$var2, df1$var1))

但由于错误而卡住了：

org.apache.spark.sql.AnalysisException: Resolved attribute(s) var1#176 missing from var2#178 in operator !Project [var2#178, levenshtein(var2#178, var1#176) AS dist#181].;;
!Project [var2#178, levenshtein(var2#178, var1#176) AS dist#181]

请告知。

Answer 1

您的错误是引用执行计划中不存在的表中的列。

添加crossJoin可以解决此问题：

dist_df <- df1 %>%
  crossJoin(df2) %>% 
  withColumn("dist", levenshtein(df1$var1, df2$var2)) 
dist_df %>% head()

  var1 var2 dist              
1  rat cat3    2
2  rat bat1    2
3  rat  dog    3
4  rat  toy    3
5  cat cat3    1
6  cat bat1    2

您可以从此处使用标准方法（How to select the first row of each group?）查找最接近的匹配项，例如：

best_matches <- dist_df %>% 
  groupBy("var2") %>% 
  agg(struct(dist_df$dist, dist_df$var1) %>% min() %>% alias("match"))

threshold <- 1  # Maximum match distance to keep

result <- best_matches %>% 
  select(
    best_matches$var2, 
    when(best_matches$match.dist <= threshold, best_matches$match.var1) %>% 
      alias("var1"))

result %>% head()

  var2 var1
1  dog <NA>
2 bat1  bat
3 cat3  cat
4  toy <NA>

请记住，这种方法效率很低。 Spark提供了更好的选项（Efficient string matching in Apache Spark），但这些尚未在SparkR中公开，而仅在sparklyr中部分实现。

如果要保留所有记录，而不论其质量如何，只需删除when：

best_matches %>% select(best_matches$var2, best_matches$match.var1) %>% head()

  var2 var1
1  dog  bat
2 bat1  bat
3 cat3  cat
4  toy  bat

SparkR：levenshtein来自2个Spark数据帧的2个变量之间的模糊字符串匹配

1 个答案: