我需要通过desc命令我的spark数据框中的特定列。 例如，这里是我想重新排序的日期时间格式
"2017-11-21T12:09:23+00:00"
"2017-11-21T12:18:55+00:00"
"2017-11-21T11:41:14+00:00"
这就是我正在做的重新排序
val windowSpec = Window.partitionBy("Source_organizationId", "Source_sourceId").orderBy(unix_timestamp($"TimeStamp", "YYYY-MM-DDThh:mm:ssTZD").cast("timestamp").desc)
val latestForEachKey = finaldf.withColumn("rank", rank().over(windowSpec))
latestForEachKey.show(false)
这是输出
+------------------+-------------------------+---------------------+---------------+-------------------------+--------------+----------+---------+---------+-------------------------+-------------------------+-----------------------+---------------------------+--------------------------+-------------------+-----------------------+--------------------+------------------------+------------+----------------------+-----------+----+
|DataPartition |TimeStamp |Source_organizationId|Source_sourceId|FilingDateTime |SourceTypeCode|DocumentId|Dcn |DocFormat|StatementDate |IsFilingDateTimeEstimated|ContainsPreliminaryData|CapitalChangeAdjustmentDate|CumulativeAdjustmentFactor|ContainsRestatement|FilingDateTimeUTCOffset|ThirdPartySourceCode|ThirdPartySourcePriority|SourceTypeId|ThirdPartySourceCodeId|FFAction|!||rank|
+------------------+-------------------------+---------------------+---------------+-------------------------+--------------+----------+---------+---------+-------------------------+-------------------------+-----------------------+---------------------------+--------------------------+-------------------+-----------------------+--------------------+------------------------+------------+----------------------+-----------+----+
|SelfSourcedPublic |2018-01-11T10:48:00+00:00|5046043341 |106 |1997-05-12T12:00:00+00:00|10Q |2426194 |null |null |1997-03-31T00:00:00+00:00|false |false |1997-03-31T00:00:00+00:00 |0.22223 |false |0 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2017-05-04T09:46:05+00:00|4295907168 |450 |2016-06-30T00:00:00+00:00|ESGWEB |null |null |null |2016-06-30T00:00:00+00:00|true |false |null |1.0 |false |0 |ATD |null |1002198005 |null |I|!| |1 |
|SelfSourcedPrivate|2017-11-02T10:27:19+00:00|4298009288 |11 |2006-10-12T12:00:00+00:00|10K |7740531 |null |ASFILED |2006-07-31T00:00:00+00:00|false |false |2006-07-31T00:00:00+00:00 |1.0 |false |0 |SS |1 |3011835 |1000716240 |I|!| |1 |
|SelfSourcedPrivate|2017-11-02T10:27:19+00:00|4298009288 |44 |2007-06-07T12:00:10+00:00|10Q |null |07905477 |ASFILED |2007-04-30T00:00:00+00:00|false |false |2007-04-30T00:00:00+00:00 |1.0 |false |0 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T10:01:29+00:00|5046043341 |97 |1999-08-12T16:00:00+00:00|10Q |null |null |null |1999-06-30T00:00:00+00:00|false |false |1999-06-30T00:00:00+00:00 |0.33334 |false |-240 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2017-11-21T12:09:23+00:00|4295904170 |364 |2017-08-08T17:00:00+00:00|10Q |null |null |null |2017-07-30T00:00:00+00:00|false |false |2017-07-30T00:00:00+00:00 |1.0 |false |-300 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2017-11-21T12:18:55+00:00|4295904170 |364 |2017-08-08T17:00:00+00:00|10Q |null |null |null |2017-07-30T00:00:00+00:00|false |false |2017-07-30T00:00:00+00:00 |1.0 |false |-300 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2017-11-21T11:41:14+00:00|4295904170 |364 |2017-08-08T17:00:00+00:00|10Q |null |null |null |2017-07-30T00:00:00+00:00|false |false |2017-07-30T00:00:00+00:00 |1.0 |false |-300 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T11:32:25+00:00|5046043341 |82 |2004-08-09T12:00:00+00:00|10Q |3932989 |null |null |2004-06-30T00:00:00+00:00|false |false |2004-06-30T00:00:00+00:00 |0.5 |false |0 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T11:20:05+00:00|5046043341 |82 |2004-08-09T12:00:00+00:00|10Q |3932989 |null |null |2004-06-30T00:00:00+00:00|false |false |2004-06-30T00:00:00+00:00 |0.5 |false |0 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T11:05:50+00:00|5046043341 |76 |2006-05-09T16:00:00+00:00|10Q |7165449 |null |ASFILED |2006-03-31T00:00:00+00:00|false |false |2006-03-31T00:00:00+00:00 |0.5 |false |-240 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T10:01:29+00:00|5046043341 |98 |1999-05-12T16:00:00+00:00|10Q |2429120 |null |null |1999-03-31T00:00:00+00:00|false |false |1999-03-31T00:00:00+00:00 |0.33334 |false |-240 |SS |1 |3011836 |1000716240 |I|!| |1 |
|Japan |2017-04-25T07:34:37+00:00|4295876606 |159 |2017-01-01T03:00:00+00:00|JPNINR |null |null |null |2016-12-31T00:00:00+00:00|false |false |2016-12-31T00:00:00+00:00 |1.0 |false |540 |SS |1 |3012418 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T11:20:05+00:00|5046043341 |80 |2005-05-09T12:00:00+00:00|10Q |4376029 |null |null |2005-03-31T00:00:00+00:00|false |false |2005-03-31T00:00:00+00:00 |0.5 |false |0 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPrivate|2017-11-02T10:27:19+00:00|4298009288 |12 |2006-10-12T12:00:10+00:00|10K |null |061140985|ASFILED |2006-07-31T00:00:00+00:00|false |false |2006-07-31T00:00:00+00:00 |1.0 |false |0 |SS |1 |3011835 |1000716240 |I|!| |1 |
|SelfSourcedPrivate|2017-11-02T10:28:44+00:00|4298009288 |29 |2010-03-10T17:00:10+00:00|10Q |null |10669005 |ASFILED |2010-01-31T00:00:00+00:00|false |false |2010-01-31T00:00:00+00:00 |1.0 |false |-300 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPrivate|2017-11-02T10:27:19+00:00|4298009288 |43 |2007-06-07T12:00:00+00:00|10Q |8753978 |null |ASFILED |2007-04-30T00:00:00+00:00|false |false |2007-04-30T00:00:00+00:00 |1.0 |false |0 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T10:48:00+00:00|5046043341 |110 |1996-05-15T12:00:00+00:00|10Q |null |null |null |1996-03-31T00:00:00+00:00|true |false |1996-03-31T00:00:00+00:00 |0.22223 |false |0 |SS |1 |3011836 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T10:48:00+00:00|5046043341 |107 |1997-03-31T12:00:00+00:00|10K |null |null |null |1996-12-31T00:00:00+00:00|true |false |1996-12-31T00:00:00+00:00 |0.22223 |false |0 |SS |1 |3011835 |1000716240 |I|!| |1 |
|SelfSourcedPublic |2018-01-11T11:05:50+00:00|5046043341 |14 |2006-03-15T17:00:00+00:00|10K |6943116 |null |ASFILED |2005-12-31T00:00:00+00:00|false |false |2005-12-31T00:00:00+00:00 |0.5 |false |-300 |SS |1 |3011835 |1000716240 |I|!| |1 |
+------------------+-------------------------+---------------------+---------------+-------------------------+--------------+----------+---------+---------+-------------------------+-------------------------+-----------------------+---------------------------+--------------------------+-------------------+-----------------------+--------------------+------------------------+------------+----------------------+-----------+----+
例如，行（4295904170 |364）与数据框中的最后一列具有相同的rank错误。
如果排序有效，则排名将变为3,2,1