我在Java 8中使用spark-sql-2.4.1v。 对于某些给定数据，我需要计算百分比，例如25,75,90。
给出源数据集：
val df = Seq(
(10, "1/15/2018", 0.010680705, 10, 0.619875458, 0.010680705, "east"),
(10, "1/15/2018", 0.006628853, 4, 0.16039063, 0.01378215, "west"),
(10, "1/15/2018", 0.01378215, 20, 0.082049528, 0.010680705, "east"),
(10, "1/15/2018", 0.810680705, 6, 0.819875458, 0.702228853, "west"),
(10, "1/15/2018", 0.702228853, 30, 0.916039063, 0.810680705, "east"),
(11, "1/15/2018", 0.010680705, 10, 0.619875458, 0.010680705, "east"),
(11, "1/15/2018", 0.006628853, 4, 0.16039063, 0.01378215, "west"),
(11, "1/15/2018", 0.01378215, 20, 0.082049528, 0.010680705, "east"),
(11, "1/15/2018", 0.810680705, 6, 0.819875458, 0.702228853, "west"),
(11, "1/15/2018", 0.702228853, 30, 0.916039063, 0.810680705, "east"))
.toDF("id", "date", "revenue", "con_dist_1", "con_dist_2", "con_dist_3", "zone")
val percentiles = Seq(0.25, 0.75,0.90) // Which percentiles to calculate
val cols = Seq("con_dist_1", "con_dist_2") // The columns to use
+---+---------+-----------+----------+-----------+-----------+----+
| id| date| revenue|con_dist_1| con_dist_2| con_dist_3|zone|
+---+---------+-----------+----------+-----------+-----------+----+
| 10|1/15/2018|0.010680705| 10|0.619875458|0.010680705|east|
| 10|1/15/2018|0.006628853| 4| 0.16039063| 0.01378215|west|
| 10|1/15/2018| 0.01378215| 20|0.082049528|0.010680705|east|
| 10|1/15/2018|0.810680705| 6|0.819875458|0.702228853|west|
| 10|1/15/2018|0.702228853| 30|0.916039063|0.810680705|east|
| 11|1/15/2018|0.010680705| 10|0.619875458|0.010680705|east|
| 11|1/15/2018|0.006628853| 4| 0.16039063| 0.01378215|west|
| 11|1/15/2018| 0.01378215| 20|0.082049528|0.010680705|east|
| 11|1/15/2018|0.810680705| 6|0.819875458|0.702228853|west|
| 11|1/15/2018|0.702228853| 30|0.916039063|0.810680705|east|
+---+---------+-----------+----------+-----------+-----------+----+
我需要为给定列的每个区域计算给定百分位数。如何实现？
预期结果
+----+---------+-------------+-------------+------------+-------------+
| id| date| revenue| perctile_col| quantile_0 |quantile_10 |
+----+---------+-------------+-------------+------------+-------------+
| 10|1/15/2018| 0.010680705| con_dist_1 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.010680705| con_dist_2 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.006628853| con_dist_1 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.006628853| con_dist_2 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.01378215| con_dist_1 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.01378215| con_dist_2 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.010680705| con_dist_1 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.010680705| con_dist_2 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.006628853| con_dist_1 |<quant0_val>|<quant10_val>|
| 10|1/15/2018| 0.006628853| con_dist_2 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.010680705| con_dist_1 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.010680705| con_dist_2 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.006628853| con_dist_1 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.006628853| con_dist_2 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.01378215| con_dist_1 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.01378215| con_dist_2 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.010680705| con_dist_1 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.010680705| con_dist_2 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.006628853| con_dist_1 |<quant0_val>|<quant10_val>|
| 11|1/15/2018| 0.006628853| con_dist_2 |<quant0_val>|<quant10_val>|
+----+---------+-------------+-------------+------------+-------------+
注意：分位数_0等在这里分别是0、50等百分位。