Question

第一个是 [ProdMetrics的最大日期来自ProdMetrics的气体总和] / 1000 第二个是（[[ProdMetrics的最大日期的ProdMetrics气体总和-[Maxd日期的ProdMetrics的气体总和-ProdMetrics的1]）/ [Maxd日期ProdMetrics的气体的总和-ProdMetrics的1]

w_maxDate = Window().orderBy(lit('A'))
prodmetrics_datasource_df = prodmetrics_datasource_df.withColumn("MaxDate", max("Date").over(w_maxDate))
prodmetrics_datasource_df = prodmetrics_datasource_df.withColumn("MaxDate", date_format("MaxDate","yyy-MM-dd"))
print("BEFORE THE FILTER")
prodmetrics_datasource_df.show(prodmetrics_datasource_df.count())
prodmetrics_datasource_df = prodmetrics_datasource_df.withColumn("OneDayAgoToMaxDate", lit(date_sub(prodmetrics_datasource_df.MaxDate,1)))
prodmetrics_datasource_df_filter_max_date = prodmetrics_datasource_df.filter(prodmetrics_datasource_df.Date == prodmetrics_datasource_df.MaxDate)
prodmetrics_datasource_df_filter_one_day_ago_to_max_date = prodmetrics_datasource_df.filter(prodmetrics_datasource_df.Date == prodmetrics_datasource_df.OneDayAgoToMaxDate)
prodmetrics_datasource_df_filter_one_day_ago_to_max_date.show()
if prodmetrics_datasource_df_filter_max_date.count() > 0:
    yesterday_production_h2 = prodmetrics_datasource_df_filter_max_date.groupby().agg(sum("AverageGas").alias("Value")/1000, sum("AverageGas")
                                                                                      .alias("SumAverageGas"))\
        .withColumn("CalcType",lit("YestProduction")).withColumn("ElementType",lit("H2")).withColumn("id",lit("YestProduction-H2"))```

([Sum of Gas from ProdMetrics on max date of ProdMetrics] - [Sum of Gas from ProdMetrics on max date - 1 of ProdMetrics]) / [Sum of Gas from ProdMetrics on max date - 1 of ProdMetrics] how can i calculate this i filtered the data

在pyspark中计算两个不同的数据框

0 个答案: