Question

我在spark中有以下架构，并希望将其展平。

root
 |-- binlog_read_timestamp: string (nullable = true)
 |-- row: struct (nullable = true)
 |    |-- after_values: struct (nullable = true)
 |    |    |-- id: long (nullable = true)
 |    |-- before_values: struct (nullable = true)
 |    |    |-- id: long (nullable = true)
 |    |-- values: struct (nullable = true)
 |    |    |-- id: long (nullable = true)
 |-- schema: string (nullable = true)
 |-- table: string (nullable = true)
 |-- type: string (nullable = true)

所以取决于type的值，我想做以下事情：

IF type == A THEN add new column with after_values.id 
IF type == B THEN add new column with before_values.id 
IF type == C THEN add new column with values.id

有关如何操作的任何建议？谢谢！

Answer 1

尝试

from pyspark.sql.functions import *

df.withColumn("new_column",
   when(col("type") == "A", col("after_values.id")) \
     .when(col("type") == "B", col("before_values.id")) \
     .when(col("type") == "C", col("values.id")))

如何在spark中展平嵌套结构

1 个答案: