pyspark的窗口函数fn.avg()仅输出相同的数据

时间:2018-07-13 01:48:14

标签: pyspark pyspark-sql

这是我的代码:

import pandas as pd
from pyspark.sql import SQLContext
import pyspark.sql.functions as fn
from pyspark.sql.functions import isnan, isnull
from pyspark.sql.functions import lit
from pyspark.sql.window import Window

spark= SparkSession.builder.appName(" ").getOrCreate()
file = "D:\project\HistoryData.csv"
lines = pd.read_csv(file)
spark_df=spark.createDataFrame(cc,['id','time','average','max','min'])

temp = Window.partitionBy("time").orderBy("id").rowsBetween(-1, 1)

df = spark_df.withColumn("movingAvg",fn.avg("average").over(temp))

df.show()

但是它输出: enter image description here

它输出相同的数据,有些数据消失了。

0 个答案:

没有答案