Question

这是我的桌子：

+--------+----+----------+----+
| station|temp|     dateS|hour|
+--------+----+----------+----+
|Roma    | 2.2|2018-10-02|   1|
|Roma    | 1.5|2018-10-02|   2|
|Roma    | 1.4|2018-10-02|   3|
|Roma    | 1.4|2018-10-02|   4|
|Milano  | 0.6|2018-11-02|  12|
|Milano  | 1.0|2018-11-02|  13|
|Napoli  | 0.3|2018-12-02|  20|
|Napoli  | 0.0|2018-12-02|  21|
|Napoli  | 1.8|2018-12-02|   4|
|Napoli  | 2.0|2018-12-03|   5|
|Napoli  | 1.8|2018-12-03|   6|
+--------+----+----------+----+

我要连续4个小时（显然是同一天连续4个小时）乘坐记录在案的电台。例如仅显示Roma，因为同一天（2018-10-02）连续4小时...

我尝试编写查询，但是什么也没有...

Answer 1

我尝试过使用引导窗口功能。我通过对桩号和dateS进行分区并按小时排序来排列数据框，并计算与前一小时的差异。如果我们连续考虑4个小时，则差异列中应该三个接一个的1。为了发现这一点，我已经根据station和dateS收集了所有差异，并检查其中是否包含“ 1 1 1”。相同的代码如下所示。希望对您有所帮助。

//Creating Test Data
  val df = Seq(("Roma",2.2,"2018-10-02",1 ) 
    , ("Roma",1.5,"2018-10-02",2 ) 
    , ("Roma",1.4,"2018-10-02",3 ) 
    , ("Roma",1.4,"2018-10-02",4 ) 
    , ("Milano",0.6,"2018-11-02",12 ) 
    , ("Milano",1.0,"2018-11-02",13 ) 
    , ("Napoli",0.3,"2018-12-02",20 ) 
    , ("Napoli",0.0,"2018-12-02",21 ) 
    , ("Napoli",1.8,"2018-12-02",4 ) 
    , ("Napoli",2.0,"2018-12-03",5 ) 
    , ("Napoli",1.8,"2018-12-03",6))
    .toDF("station", "temp", "dateS", "hour")

  val filterDF = df.withColumn("hour_lead", lead($"hour", 1)
                   .over(Window.partitionBy("station","dateS")
                               .orderBy(col("hour")))
                   .filter($"hour_lead".isNotNull)
                   .withColumn("hour_diff", $"hour_lead" - $"hour")
                   .groupBy("station","dateS")
          .agg(collect_list($"hour_diff".cast("string")).as("hour_diff_list"))
          .withColumn("hour_diff_list_str", 
                      concat(lit(" "), 
                      concat_ws(" ", $"hour_diff_list"), 
                      lit(" ")))
          .filter($"hour_diff_list_str".contains(" 1 1 1 "))

  filterDF.show(false)
  +-------+----------+--------------+------------------+
  |station|dateS     |hour_diff_list|hour_diff_list_str|
  +-------+----------+--------------+------------------+
  |Roma   |2018-10-02|[1, 1, 1]     | 1 1 1            |
  +-------+----------+--------------+------------------+

Spark SQL需要连续4个时间间隔

1 个答案: