Spark SQL-数据比较

Question

比较具有相同模式和主键列的两个csv文件（数百万行）并打印出差异的最佳方法是什么。例如

CSV1

Id  name   zip   
1   name1  07112  
2   name2  07234  
3   name3  10290  

CSV2

Id  name    zip   
1   name1   07112  
2   name21  07234  
4   name4   10290  

将修改后的文件CSV2与原始数据CSV1比较，

输出应为

Id name    zip   
2  name21  07234 Modified  
3  name3   10290 Deleted  
4  name4   10290 Added  

Spark SQL的新手，我正在考虑将数据导入Hive表，然后运行Spark SQL来识别更改。

1）是否有可用的行修改方法来标识行是否已修改，而不是比较每列中的值？ 2）是否有更好的方法可用于使用Spark或其他HDFS工具？

赞赏反馈

Answer 1

存在许多方法；这是可以并行完成的任务：

import org.apache.spark.sql.functions._
import sqlContext.implicits._

val origDF = sc.parallelize(Seq(
  ("1", "a", "b"),
  ("2", "c", "d"),
  ("3", "e", "f")
)).toDF("k", "v1", "v2")

val newDF = sc.parallelize(Seq(
  ("1", "a", "b"),
  ("2", "c2", "d"),
  ("4", "g", "h")
)).toDF("k", "v1", "v2")

val df1 = origDF.except(newDF) // if k not exists in df2, then deleted
//df1.show(false)
val df2 = newDF.except(origDF) // if k not exists in df1, then added
//df2.show(false)

                           // if no occurrence in both dfs, then the same
                           // if k exists in both, then k in df2 = modified

df1.createOrReplaceTempView("df1")
df2.createOrReplaceTempView("df2")

val df3 = spark.sql("""SELECT df1.k, df1.v1, df1.v2, "deleted" as operation
                         FROM  df1 
                        WHERE NOT EXISTS (SELECT df2.k 
                                            FROM df2
                                            WHERE df2.k = df1.k)
                          UNION
                       SELECT df2.k, df2.v1, df2.v2, "added" as operation
                         FROM  df2 
                         WHERE NOT EXISTS (SELECT df1.k 
                                             FROM df1
                                            WHERE df1.k = df2.k)
                          UNION
                       SELECT df2.k, df2.v1, df2.v2, "modified" as operation
                         FROM  df2 
                        WHERE EXISTS (SELECT df1.k 
                                        FROM df1
                                        WHERE df1.k = df2.k)

                   """)

df3.show(false)

返回：

+---+---+---+---------+
|k  |v1 |v2 |operation|
+---+---+---+---------+
|4  |g  |h  |added    |
|2  |c2 |d  |modified |
|3  |e  |f  |deleted  |
+---+---+---+---------+

不太困难，没有标准实用程序。