在Spark结构化流中创建嵌套列

Question

如何在Spark Streaming中将列转换为嵌套列？

实际：

root
 |-- channelId: string (nullable = true)
 |-- country: string (nullable = true)
 |-- product: string (nullable = true)
 |-- sourceId: string (nullable = true)
 |-- systemId: string (nullable = true)
 |-- destinationId: string (nullable = true)

预期：

root
|-- txn_summary: struct (nullable = true)
|   |-- channelId: string (nullable = true)
|   |-- country: string (nullable = true)
|   |-- product: string (nullable = true)
|   |-- sourceId: string (nullable = true)
|   |-- systemId: string (nullable = true)
|   |-- destinationId: string (nullable = true)

Answer 1

您可以将结构function用作

import org.apache.spark.sql.functions.*;
dataframe.select(struct(dataframe.columns()).as("txn_summary"));