Question

我发现很难在DF中转置列。下面给出的是基本数据帧和预期的输出

Student    Class         Subject      Grade    
Sam        6th Grade     Maths        A
Sam        6th Grade     Science      A
Sam        7th Grade     Maths        A-
Sam        7th Grade     Science      A
Rob        6th Grade     Maths        A
Rob        6th Grade     Science      A-
Rob        7th Grade     Maths        A-
Rob        7th Grade     Science      B
Rob        7th Grade     AP           A

预期输出：

Student Class        Math_Grade  Science_Grade  AP_Grade
Sam     6th Grade    A           A  
Sam     7th Grade    A-          A  
Rob     6th Grade    A           A- 
Rob     7th Grade    A-          B               A

请提出解决此问题的最佳方法。

Answer 1

您可以通过group Student, Class来使用DataFrame并按如下所示枢转Subject：

import org.apache.spark.sql.functions._

val df = Seq(
  ("Sam", "6th Grade", "Maths", "A"),
  ("Sam", "6th Grade", "Science", "A"),
  ("Sam", "7th Grade", "Maths", "A-"),
  ("Sam", "7th Grade", "Science", "A"),
  ("Rob", "6th Grade", "Maths", "A"),
  ("Rob", "6th Grade", "Science", "A-"),
  ("Rob", "7th Grade", "Maths", "A-"),
  ("Rob", "7th Grade", "Science", "B"),
  ("Rob", "7th Grade", "AP", "A")
).toDF("Student", "Class", "Subject", "Grade")

df.
  groupBy("Student", "Class").pivot("Subject").agg(first("Grade")).
  orderBy("Student", "Class").
  show
// +-------+---------+----+-----+-------+
// |Student|    Class|  AP|Maths|Science|
// +-------+---------+----+-----+-------+
// |    Rob|6th Grade|null|    A|     A-|
// |    Rob|7th Grade|   A|   A-|      B|
// |    Sam|6th Grade|null|    A|      A|
// |    Sam|7th Grade|null|   A-|      A|
// +-------+---------+----+-----+-------+

Answer 2

只需使用pivot and group based on columns。

 case class StudentRecord(Student: String, `Class`: String, Subject: String, Grade: String)

 val rows = Seq(StudentRecord
  ("Sam", "6th Grade", "Maths", "A"),
  StudentRecord
  ("Sam", "6th Grade", "Science", "A"),
  StudentRecord
  ("Sam", "7th Grade", "Maths", "A-"),
  StudentRecord
  ("Sam", "7th Grade", "Science", "A"),
  StudentRecord
  ("Rob", "6th Grade", "Maths", "A"),
  StudentRecord
  ("Rob", "6th Grade", "Science", "A-"),
  StudentRecord
  ("Rob", "7th Grade", "Maths", "A-"),
  StudentRecord
  ("Rob", "7th Grade", "Science", "B"),
  StudentRecord
  ("Rob", "7th Grade", "AP", "A")
).toDF()

 rows.groupBy("Student", "Class").pivot("Subject").agg(first("Grade")).orderBy(desc("Student"), asc("Class")).show()


 /**
  * +-------+---------+----+-----+-------+
  * |Student|    Class|  AP|Maths|Science|
  * +-------+---------+----+-----+-------+
  * |    Sam|6th Grade|null|    A|      A|
  * |    Sam|7th Grade|null|   A-|      A|
  * |    Rob|6th Grade|null|    A|     A-|
  * |    Rob|7th Grade|   A|   A-|      B|
  * +-------+---------+----+-----+-------+
  */

在Spark Scala中转置DataFrame列

2 个答案: