我有一个Dataframe,希望将它分成相同数量的行。
换句话说,我想要一个数据帧列表,其中每个数据帧都是原始数据帧的脱节子集。
让我们说输入dataframer如下:
+------------------+-----------+-----+--------------------+
| eventName|original_dt|count| features|
+------------------+-----------+-----+--------------------+
|15.509775004326936| 0| 100|[15.5097750043269...|
|15.509775004326936| 0| 101|[15.5097750043269...|
|15.509775004326936| 0| 102|[15.5097750043269...|
|15.509775004326936| 0| 103|[15.5097750043269...|
|15.509775004326936| 0| 104|[15.5097750043269...|
|15.509775004326936| 0| 105|[15.5097750043269...|
|15.509775004326936| 0| 106|[15.5097750043269...|
|15.509775004326936| 0| 107|[15.5097750043269...|
|15.509775004326936| 0| 108|[15.5097750043269...|
|15.509775004326936| 0| 109|[15.5097750043269...|
|15.509775004326936| 0| 110|[15.5097750043269...|
|15.509775004326936| 0| 111|[15.5097750043269...|
|15.509775004326936| 0| 112|[15.5097750043269...|
|15.509775004326936| 0| 113|[15.5097750043269...|
|15.509775004326936| 0| 114|[15.5097750043269...|
|15.509775004326936| 0| 115|[15.5097750043269...|
| 43.01955000865387| 0| 116|[43.0195500086538...|
+------------------+-----------+-----+--------------------+
我希望将其拆分为K个相等大小的数据帧。如果k = 4,则可能的结果是:
+------------------+-----------+-----+--------------------+
| eventName|original_dt|count| features|
+------------------+-----------+-----+--------------------+
|15.509775004326936| 0| 106|[15.5097750043269...|
|15.509775004326936| 0| 107|[15.5097750043269...|
|15.509775004326936| 0| 110|[15.5097750043269...|
|15.509775004326936| 0| 111|[15.5097750043269...|
+------------------+-----------+-----+--------------------+
+------------------+-----------+-----+--------------------+
| eventName|original_dt|count| features|
+------------------+-----------+-----+--------------------+
|15.509775004326936| 0| 104|[15.5097750043269...|
|15.509775004326936| 0| 108|[15.5097750043269...|
|15.509775004326936| 0| 112|[15.5097750043269...|
|15.509775004326936| 0| 114|[15.5097750043269...|
+------------------+-----------+-----+--------------------+
+------------------+-----------+-----+--------------------+
| eventName|original_dt|count| features|
+------------------+-----------+-----+--------------------+
|15.509775004326936| 0| 100|[15.5097750043269...|
|15.509775004326936| 0| 105|[15.5097750043269...|
|15.509775004326936| 0| 109|[15.5097750043269...|
|15.509775004326936| 0| 115|[15.5097750043269...|
+------------------+-----------+-----+--------------------+
+------------------+-----------+-----+--------------------+
| eventName|original_dt|count| features|
+------------------+-----------+-----+--------------------+
|15.509775004326936| 0| 101|[15.5097750043269...|
|15.509775004326936| 0| 102|[15.5097750043269...|
|15.509775004326936| 0| 103|[15.5097750043269...|
|15.509775004326936| 0| 113|[15.5097750043269...|
| 43.01955000865387| 0| 116|[43.0195500086538...|
+------------------+-----------+-----+--------------------+
答案 0 :(得分:2)
根据我对您的输入和所需输出的理解,您可以row numbers
grouping
dataframe
创建one groupId
。
然后,您可以filter
dataframe
根据您的需要在其他地方比较row number
和storing
。
以下是满足您需求的临时解决方案。您可以根据需要进行更改
val k = 4
val windowSpec = Window.partitionBy("grouped").orderBy("original_dt")
val newDF = dataFrame.withColumn("grouped", lit("grouping"))
var latestDF = newDF.withColumn("row", row_number() over windowSpec)
val totalCount = latestDF.count()
var lowLimit = 0
var highLimit = lowLimit + k
while(lowLimit < totalCount){
latestDF.where(s"row <= ${highLimit} and row > ${lowLimit}").show(false)
lowLimit = lowLimit + k
highLimit = highLimit + k
}
我希望这会给你一个良好的开端。
答案 1 :(得分:1)
另一种解决方案是使用限制和除外。以下程序将返回一个数组具有相同行数的数组。除了可能包含较少行的第一个。
var numberOfNew = 4
var input = List(1,2,3,4,5,6,7,8,9).toDF
var newFrames = 0 to numberOfNew map (_ => Seq.empty[Int].toDF) toArray
var size = input.count();
val limit = (size / numberOfNew).toInt
while (size > 0) {
newFrames(numberOfNew) = input.limit(limit)
input = input.except(newFrames(numberOfNew))
size = size - limit
numberOfNew = numberOfNew - 1
}
newFrames.foreach(_.show)
+-----+
|value|
+-----+
| 7|
+-----+
+-----+
|value|
+-----+
| 4|
| 8|
+-----+
+-----+
|value|
+-----+
| 5|
| 9|
+-----+
...
答案 2 :(得分:1)
这是对斯蒂芬·施密茨(Steffen Schmitz)的改进答案,实际上这是不正确的。我已经对其进行了改进,以使其后代化。但是,我确实对大规模的性能感到好奇。
var numberOfNew = 4
var input = Seq((1,2),(3,4),(5,6),(7,8),(9,10),(11,12)).toDF
var newFrames = 0 to numberOfNew-1 map (_ => Seq.empty[(Int, Int)].toDF) toArray
var size = input.count();
val limit = (size / numberOfNew).toInt
val limit_fract = (size / numberOfNew.toFloat)
val residual = ((limit_fract.toDouble - limit.toDouble) * size).toInt
var limit_to_use = limit
while (numberOfNew > 0) {
if (numberOfNew == 1 && residual != 0) limit_to_use = residual
newFrames(numberOfNew-1) = input.limit(limit_to_use)
input = input.except(newFrames(numberOfNew-1))
size = size - limit
numberOfNew = numberOfNew - 1
}
newFrames.foreach(_.show)
val singleDF = newFrames.reduce(_ union _)
singleDF.show(false)
返回单个数据帧:
+---+---+
| _1| _2|
+---+---+
| 7| 8|
| 3| 4|
| 11| 12|
+---+---+
+---+---+
| _1| _2|
+---+---+
| 5| 6|
+---+---+
+---+---+
| _1| _2|
+---+---+
| 9| 10|
+---+---+
+---+---+
| _1| _2|
+---+---+
| 1| 2|
+---+---+
答案 3 :(得分:0)
如果您要将一个数据集划分为n个相等的数据集
double[] arraySplit = {1,1,1...,n}; //you can also divide into ratio if you change the numbers.
List<Dataset<String>> datasetList = dataset.randomSplitAsList(arraySplit,1);