Question

我有一个火花数据帧（df），包含2列（Report_id和Cluster_number）。

我想将一个函数（getClusterInfo）应用于df，它将返回每个集群的名称，即如果集群号是＆＃39; 3＆＃39;然后对于特定的report_id，将写下下面提到的3行：

{"cluster_id":"1","influencers":[{"screenName":"A"},{"screenName":"B"},{"screenName":"C"},...]}
{"cluster_id":"2","influencers":[{"screenName":"D"},{"screenName":"E"},{"screenName":"F"},...]}
{"cluster_id":"3","influencers":[{"screenName":"G"},{"screenName":"H"},{"screenName":"E"},...]}

我在df上使用foreach来应用getClusterInfo功能，但无法弄清楚如何将o / p转换为数据帧（Report_id，{{1} }）。

以下是代码段：

Array[cluster_info]

Answer 1

一般来说，当您希望将某些内容映射到其他内容时，您不应该使用foreach; foreach适用于仅具有副作用并且不返回任何内容的函数。

在这种情况下，如果我得到了正确的详细信息（可能没有），您可以使用用户定义函数（UDF）和explode结果：

import org.apache.spark.sql.functions._
import spark.implicits._

// I'm assuming we have these case classes (or similar)
case class Influencer(screenName: String)
case class ClusterInfo(cluster_id: String, influencers: Array[Influencer])

// I'm assuming this method is supplied - with your own implementation
def getClusterInfo(clusterId: String): ClusterInfo =
  ClusterInfo(clusterId, Array(Influencer(clusterId)))

// some sample data - assuming both columns are integers:
val df = Seq((222, 3), (333, 4)).toDF("Report_id", "Cluster_number")

// actual solution:

// UDF that returns an array of ClusterInfo;
// Array size is 'clusterNo', creates cluster id for each element and maps it to info
val clusterInfoUdf = udf { (clusterNo: Int, reportId: Int) =>
  (1 to clusterNo).map(v => s"${reportId}_$v").map(getClusterInfo)
}

// apply UDF to each record and explode - to create one record per array item
val result = df.select(explode(clusterInfoUdf($"Cluster_number", $"Report_id")))

result.printSchema()
// root
// |-- col: struct (nullable = true)
// |    |-- cluster_id: string (nullable = true)
// |    |-- influencers: array (nullable = true)
// |    |    |-- element: struct (containsNull = true)
// |    |    |    |-- screenName: string (nullable = true)

result.show(truncate = false)
// +-----------------------------+
// |col                          |
// +-----------------------------+
// |[222_1,WrappedArray([222_1])]|
// |[222_2,WrappedArray([222_2])]|
// |[222_3,WrappedArray([222_3])]|
// |[333_1,WrappedArray([333_1])]|
// |[333_2,WrappedArray([333_2])]|
// |[333_3,WrappedArray([333_3])]|
// |[333_4,WrappedArray([333_4])]|
// +-----------------------------+

函数到Spark Dataframe

1 个答案: