Question

我在spark中有两个数据框。我正在做df1.except(df2)两个查找是否有任何列在两个数据框之间发生了变化。

df1就像这里

|001000900|aaaaa BELLOWS CORPORATION||N|
|001000905|ddddd DEPARTMENT OF LABOR AND EMPLOYMENT SECURITY|BUREAU OF COMPLIANCE|N|
|001001049|gggg RAVIOLI MFG CO INC|SPINELLI BKY RAVIOLI PASTRY SP|N|
|001001130|dddd ANGELES UNIFIED SCHOOL DISTRICT|TRANSPORTATION BRANCH|N|
|001001143|ffff MUSIC PARTIES, INC||N|
|001001155|BOSTON BRASS AND IRON CO||N|
|001001171|HANCOCK MARINE, INC.||N|
|001001184|TRILLION CORPORATION||N|
|001001192|HAWAII STATE CHIROPRACTIC ASSOCIATION INC||N|
|001001379|THE FRUIT SQUARE PEOPLE INC|L & M BAKERY|N|
|001001416|J & S MARKET||N|

df2如下所示

|001000145|PARADISE TAN||N|
|001000306|SHRUT & ASCH LEATHER COMPANY, INC.||N|
|001000355|HARRISON SPECIALTY CO., INC.||N|
|001000363|LOUIS M. GERSON CO., INC.||N|
|001000467|SAVE THE SEA TURTLES INTERNATIONAL|ADOPT THE BEACH HI|N|
|001000504|DIRIGO SPICE CORPORATION|CUNNINGHAM SPICE|N|
|001000744|FREEDMAN THREAD COMPANY|COLONIAL THREAD CO|N|
|001000756|AFFORDABLE AIR CONDITIONING|P R ENTERPRISE|N|
|001000900|CLIFLEX BELLOWS CORPORATION||N|
|001000905|FLORIDA DEPARTMENT OF LABOR AND EMPLOYMENT SECURITY|BUREAU OF COMPLIANCE|N|
|001001049|SPINELLI RAVIOLI MFG CO INC|SPINELLI BKY RAVIOLI PASTRY SP|N|
|001001130|LOS ANGELES UNIFIED SCHOOL DISTRICT|TRANSPORTATION BRANCH|N|
|001001143|TOSCO MUSIC PARTIES, INC||N|
|001001155|BOSTON BRASS AND IRON CO||N|

但我想要的是我必须在一个列的基础上找到两个数据帧之间的差异。如下所示

我想要我的输出如下

|dunsnumber|filler1|        businessname|      tradestylename|registeredaddressindicator|
+----------+-------+--------------------+--------------------+--------------------------+
| 001001130|       |dddd ANGELES UNIF...|TRANSPORTATION BR...|                         N|
| 001000900|       |aaaaa BELLOWS COR...|                    |                         N|
| 001000905|       |ddddd DEPARTMENT ...|BUREAU OF COMPLIANCE|                         N|
| 001001143|       |ffff MUSIC PARTIE...|                    |                         N|
| 001001049|       |gggg RAVIOLI MFG ...|SPINELLI BKY RAVI...|                         N|
+----------+-------+--------------------+--------------------+

这是我的代码

    import org.apache.spark.sql.functions._
    val textRdd1 = sc.textFile("/home/cloudera/TRF/PCFP/INCR")
    val rowRdd1 = textRdd1.map(line => Row.fromSeq(line.split("\\|", -1)))
    var df1 = sqlContext.createDataFrame(rowRdd1, schema)

    val textRdd2 = sc.textFile("/home/cloudera/TRF/PCFP/MAIN")
    val rowRdd2 = textRdd2.map(line => Row.fromSeq(line.split("\\|", -1)))
    var df2 = sqlContext.createDataFrame(rowRdd2, schema)
    val diffAnyColumnDF = df1.except(df2).where(df1.col("dunsnumber") === 
    df2.col("dunsnumber")).show()

因此，如果我的主键是“密码”，那就是＆＃39;匹配然后只查找是否有任何列已更改该主键。

我希望我的问题清楚。

Answer 1

Dataframe没有substract方法。您可以使用替代方法。将数据转换为RDD，使用减法方法，返回数据帧。

Answer 2

嗨，这对我有用..

val diffAnyColumnDF = df1.except(df2)
val addDF= diffAnyColumnDF.join(df2, Seq("dunsnumber")).show()

基于spark scala中的主键在两个数据帧之间找到差异

2 个答案: