Question

您好，我是Spark和Scala的新手，我想拆分以下数据框：

df:
+----------+-----+------+----------+--------+
|        Ts| Temp|  Wind|  Precipit|Humidity|
+----------+-----+------+----------+--------+
|1579647600|   10|    22|        10|      50|
|1579734000|   11|    21|        10|      55|
|1579820400|   10|    18|        15|      60|
|1579906800|    9|    23|        20|      60|
|1579993200|    8|    24|        25|      50|
|1580079600|   10|    18|        27|      60|
|1580166000|   11|    20|        30|      50|
|1580252400|   12|    17|        15|      50|
|1580338800|   10|    14|        21|      50|
|1580425200|    9|    16|        25|      60|
-----------+-----+------+----------+--------+

结果数据框应如下：

df1:
+----------+-----+------+----------+--------+
|        Ts| Temp|  Wind|  Precipit|Humidity|
+----------+-----+------+----------+--------+
|1579647600|   10|    22|        10|      50|
|1579734000|   11|    21|        10|      55|
|1579820400|   10|    18|        15|      60|
|1579906800|    9|    23|        20|      60|
|1579993200|    8|    24|        25|      50|
|1580079600|   10|    18|        27|      60|
|1580166000|   11|    20|        30|      50|
|1580252400|   12|    17|        15|      50|
+----------+-----+------+----------+--------+
df2:
+----------+-----+------+----------+--------+
|        Ts| Temp|  Wind|  Precipit|Humidity|
+----------+-----+------+----------+--------+
|1580338800|   10|    14|        21|      50|
|1580425200|    9|    16|        25|      60|
-----------+-----+------+----------+--------+

其中df1在df的前几行中占80％，而df2的左行则占20％。

Answer 1

假设数据是随机分割的：

val Array(df1, df2) = df.randomSplit(Array(0.8, 0.2))

但是，如果用“顶部行”来表示示例数据框中的“ Ts”列，则可以这样做：

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions.{col,percent_rank}

val window = Window.partitionBy().orderBy(df['Ts'].desc())

val df1 = df.select('*', percent_rank().over(window).alias('rank')) 
  .filter(col('rank') >= 0.2) 
  .show()

val df2 = df.select('*', percent_rank().over(window).alias('rank')) 
  .filter(col('rank') < 0.2) 
  .show()

Answer 2

尝试使用 monotonically_increasing_id() 功能和 window percent_rank() ，因为此功能可保留顺序。

Example:

val df=sc.parallelize(Seq((1579647600,10,22,10,50),
(1579734000,11,21,10,55),
(1579820400,10,18,15,60),
(1579906800, 9,23,20,60),
(1579993200, 8,24,25,50),
(1580079600,10,18,27,60),
(1580166000,11,20,30,50),
(1580252400,12,17,15,50),
(1580338800,10,14,21,50),
(1580425200, 9,16,25,60)),10).toDF("Ts","Temp","Wind","Precipit","Humidity")

import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions._

val df1=df.withColumn("mid",monotonically_increasing_id)
val df_above_80=df1.withColumn("pr",percent_rank().over(w)).filter(col("pr") >= 0.8).drop(Seq("mid","pr"):_*)
val df_below_80=df1.withColumn("pr",percent_rank().over(w)).filter(col("pr") < 0.8).drop(Seq("mid","pr"):_*)

df_below_80.show()
/*
+----------+----+----+--------+--------+
|        Ts|Temp|Wind|Precipit|Humidity|
+----------+----+----+--------+--------+
|1579647600|  10|  22|      10|      50|
|1579734000|  11|  21|      10|      55|
|1579820400|  10|  18|      15|      60|
|1579906800|   9|  23|      20|      60|
|1579993200|   8|  24|      25|      50|
|1580079600|  10|  18|      27|      60|
|1580166000|  11|  20|      30|      50|
|1580252400|  12|  17|      15|      50|
+----------+----+----+--------+--------+
*/

df_above_80.show()
/*
+----------+----+----+--------+--------+
|        Ts|Temp|Wind|Precipit|Humidity|
+----------+----+----+--------+--------+
|1580338800|  10|  14|      21|      50|
|1580425200|   9|  16|      25|      60|
+----------+----+----+--------+--------+
*/

如何基于原始数据帧中的行总数将数据帧拆分为两个数据帧

2 个答案: