Question

我有一个这样的数据框：

+-----------------+------------------+-----------+--------+---+
| conversation_id |   message_body   | timestamp | sender |   |
+-----------------+------------------+-----------+--------+---+
| A               | hi               | 9:00      | John   |   |
| A               | how are you?     | 10:00     | John   |   |
| A               | can we meet?     | 10:05     | John   | * |
| A               | not bad          | 10:30     | Steven | * |
| A               | great            | 10:40     | John   |   |
| A               | yeah, let's meet | 10:35     | Steven |   |
| B               | Hi               | 12:00     | Anna   | * |
| B               | Hello            | 12:05     | Ken    | * |
+-----------------+------------------+-----------+--------+---+

对于每个对话，我都希望在第一个发送者的第一块和第二个发送者的第一块有最后一条消息。我用星号标记了它们。

我的一个想法是为第一个用户分配0，为第二个用户分配1。

理想情况下，我想拥有这样的东西：

+-----------------+---------+------------+--------------+---------+------------+----------+
| conversation_id | sender1 | timestamp1 |   message1   | sender2 | timestamp2 | message2 |
+-----------------+---------+------------+--------------+---------+------------+----------+
| A               | John    | 10:05      | can we meet? | Steven  | 10:30      | not bad  |
| B               | Anna    | 12:00      | Hi           | Ken     | 12:05      | Hello    |
+-----------------+---------+------------+--------------+---------+------------+----------+

如何在Spark中做到这一点？

Answer 1

有趣的问题出现了。

由10:35修改为10:45
使用前导0格式，例如09:00，而不是9:00
您将需要相应地使用自己的数据类型，这只是在演示所需的方法

在DataBricks Notebook中完成

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._ 
import spark.implicits._

val df = 
     Seq(("A", "hi", "09:00", "John"), ("A", "how are you?", "10:00", "John"), 
         ("A", "can we meet?", "10:05", "John"), ("A", "not bad", "10:30", "Steven"), 
         ("A", "great", "10:40", "John"), ("A", "yeah, let's meet", "10:45", "Steven"),
         ("B", "Hi", "12:00", "Anna"), ("B", "Hello", "12:05", "Ken")
        ).toDF("conversation_id", "message_body", "timestampX", "sender")

// Get rank, 1 is who were initiates conversation, the other values can be used to infer relationships
// Note no @Transient required here with Window 
val df2 = df.withColumn("rankC", row_number().over(Window.partitionBy($"conversation_id").orderBy($"timestampX".asc)))

// A value <> 1 is the first message of second Sender.
// The 1 value of this is the last message of first "block"
val df3 = df2.select('conversation_id as "df3_conversation_id", 'sender as "df3_sender", 'rankC as "df3_rank")
// To avoid pipelined renaming issues that occur
val df3a = df3.groupBy("df3_conversation_id", "df3_sender").agg(min("df3_rank") as "rankC2").filter("rankC2 != 1")

//JOIN the values with some smarts. Some odd errors in Spark thru pipe-lining gotten. Need to drop pipelined row(), ranking etc.
val df4 = df3a.join(df2, (df3a("df3_conversation_id") === df2("conversation_id")) && (df3a("rankC2") === df2("rankC") + 1)).drop("rankC").drop("rankC2")   
val df4a = df3a.join(df2, (df3a("df3_conversation_id") === df2("conversation_id")) && (df3a("rankC2") === df2("rankC"))).drop("rankC").drop("rankC2")  

// The get other missing data, could have all been combined but done in steps for simplicity. Just a simple final JOIN and you ahve the answer.
val df5 = df4.join(df4a, (df4("df3_conversation_id") === df4a("df3_conversation_id")))  
df5.show(false)

返回：

此处的输出将不完全格式化，请在REPL中运行以查看标题。

 |B                  |Ken       |B              |Hi          |12:00     |Anna  |B                  |Ken       |B              |Hello       |12:05     |Ken   |
 |A                  |Steven    |A              |can we meet?|10:05     |John  |A                  |Steven    |A              |not bad     |10:30     |Steven|

您可以进一步处理数据，现在就完成了繁重的工作。 Catalyst Optimizer在编译等方面存在一些问题，所以这就是为什么我以这种方式进行解决。

每个分组从Spark数据框中选择特定的行

1 个答案: