熊猫发现一年前的日期是非统一的日期列表

时间:2017-09-12 13:53:54

标签: python python-2.7 pandas date datetime

我可以在项目中使用更多帮助。我试图分析450万行数据。我已将数据读入数据框,已组织数据,现在有3列:1)日期为datetime 2)唯一标识符3)价格

我需要计算每件商品价格的年度变化,但日期不统一且每件商品不一致。例如:

date      item  price
12/31/15   A     110
12/31/15   B     120
12/31/14   A     100
6/24/13    B     100

我希望找到的结果是:

date      item  price  previousdate   % change
12/31/15   A     110   12/31/14       10%
12/31/15   B     120   6/24/13        20%
12/31/14   A     100
6/24/13    B     100

编辑 - 更好的数据示例

 date   item    price
6/1/2016    A   276.3457646
6/1/2016    B   5.044165645
4/27/2016   B   4.91300186
4/27/2016   A   276.4329163
4/20/2016   A   276.9991265
4/20/2016   B   4.801263717
4/13/2016   A   276.1950213
4/13/2016   B   5.582923328
4/6/2016    B   5.017863509
4/6/2016    A   276.218649
3/30/2016   B   4.64274783
3/30/2016   A   276.554653
3/23/2016   B   5.576438253
3/23/2016   A   276.3135836
3/16/2016   B   5.394435443
3/16/2016   A   276.4222986
3/9/2016    A   276.8929462
3/9/2016    B   4.999951262
3/2/2016    B   4.731349423
3/2/2016    A   276.3972068
1/27/2016   A   276.8458971
1/27/2016   B   4.993033132
1/20/2016   B   5.250379701
1/20/2016   A   276.2899864
1/13/2016   B   5.146639666
1/13/2016   A   276.7041978
1/6/2016    B   5.328296958
1/6/2016    A   276.9465891
12/30/2015  B   5.312301356
12/30/2015  A   256.259668
12/23/2015  B   5.279105491
12/23/2015  A   255.8411198
12/16/2015  B   5.150798234
12/16/2015  A   255.8360529
12/9/2015   A   255.4915183
12/9/2015   B   4.722876886
12/2/2015   A   256.267146
12/2/2015   B   5.083626167
10/28/2015  B   4.876177757
10/28/2015  A   255.6464653
10/21/2015  B   4.551439655
10/21/2015  A   256.1735769
10/14/2015  A   255.9752668
10/14/2015  B   4.693967392
10/7/2015   B   4.911797443
10/7/2015   A   256.2556707
9/30/2015   B   4.262994526
9/30/2015   A   255.8068691
7/1/2015    A   255.7312385
4/22/2015   A   234.6210132
4/15/2015   A   235.3902076
4/15/2015   B   4.154926102
4/1/2015    A   234.4713827
2/25/2015   A   235.1391496
2/18/2015   A   235.1223471

我所做的事情(在其他用户的帮助下)并没有起作用,但却在下面。感谢您提供的任何帮助,或指出我正确的方向!

import pandas as pd
import datetime as dt
import numpy as np

df = pd.read_csv('...python test file5.csv',parse_dates =['As of Date'])

df = df[['item','price','As of Date']]

def get_prev_year_price(x, df):
    try:
        return df.loc[x['prev_year_date'], 'price']
        #return np.abs(df.time - x)
    except Exception as e:
        return x['price']

#Function to determine the closest date from given date and list of all dates
def nearest(items, pivot):
    return min(items, key=lambda x: abs(x - pivot))

df['As of Date'] = pd.to_datetime(df['As of Date'],format='%m/%d/%Y')
df = df.rename(columns = {df.columns[2]:'date'})

# list of dates
dtlst = [item for item in df['date']]

data = []
data2 = []
for item in df['item'].unique():
    item_df = df[df['item'] == item] #select based on items
    select_dates = item_df['date'].unique()
    item_df.set_index('date', inplace=True) #set date as key index

    item_df = item_df.resample('D').mean().reset_index() #fill in missing date
    item_df['price'] = item_df['price'].interpolate('nearest') #fill in price with nearest price available
    # use max(item_df['date'] where item_df['date'] < item_df['date'] - pd.DateOffset(years=1, days=1))
        #possible_date = item_df['date'] - pd.DateOffset(years=1)
        #item_df['prev_year_date'] = max(df[df['date'] <= possible_date])

    item_df['prev_year_date'] = item_df['date'] - pd.DateOffset(years=1) #calculate 1 year ago date
    date_df = item_df[item_df.date.isin(select_dates)] #select dates with useful data
    item_df.set_index('date', inplace=True)

    date_df['prev_year_price'] = date_df.apply(lambda x: get_prev_year_price(x, item_df),axis=1)
    #date_df['prev_year_price'] = date_df.apply(lambda x: nearest(dtlst, x),axis=1)

    date_df['change'] = date_df['price'] / date_df['prev_year_price']-1
    date_df['item'] = item
    data.append(date_df)
    data2.append(item_df)
summary = pd.concat(data).sort_values('date', ascending=False)
#print (summary)

#saving the output of the CSV file to see how data looks after being handled 
filename = '...python_test_file_save4.csv'
summary.to_csv(filename, index=True, encoding='utf-8')

2 个答案:

答案 0 :(得分:2)

使用当前的用例假设,这适用于此特定用例

 <p-checkbox [value]="item1" formControlName="selectedComponents" [label]="item1.itemSku"></p-checkbox>

排序In [2459]: def change(grp): ...: grp['% change'] = grp.price.diff() ...: grp['previousdate'] = grp.date.shift(1) ...: return grp 然后dategroupby apply函数,然后对索引进行排序。

change

答案 1 :(得分:1)

merge_asof这是一个很好的情况,它通过查找右数据帧的最后一行(小于左数据帧的键)来合并两个数据帧。我们需要首先在正确的数据框架中添加一年,因为日期之间的要求是1年或更长时间。

以下是您在评论中提到的一些示例数据。

date      item  price
12/31/15   A     110
12/31/15   B     120
12/31/14   A     100
6/24/13    B     100
12/31/15   C     100
1/31/15    C      80
11/14/14   C     130
11/19/13   C     110
11/14/13   C     200

日期需要按merge_asof排序才能生效。 merge_asof也会删除加入列,因此我们需要将其副本放回正确的数据框中。

设置数据框

df = df.sort_values('date')
df_copy = df.copy()
df_copy['previousdate'] = df_copy['date']
df_copy['date'] += pd.DateOffset(years=1)

使用merge_asof

df_final = pd.merge_asof(df, df_copy, 
                        on='date', 
                        by='item', 
                        suffixes=['current', 'previous'])
df_final['% change'] = (df_final['pricecurrent'] - df_final['priceprevious']) / df_final['priceprevious']
df_final

        date item  pricecurrent  priceprevious previousdate  % change
0 2013-06-24    B           100            NaN          NaT       NaN
1 2013-11-14    C           200            NaN          NaT       NaN
2 2013-11-19    C           110            NaN          NaT       NaN
3 2014-11-14    C           130          200.0   2013-11-14 -0.350000
4 2014-12-31    A           100            NaN          NaT       NaN
5 2015-01-31    C            80          110.0   2013-11-19 -0.272727
6 2015-12-31    A           110          100.0   2014-12-31  0.100000
7 2015-12-31    B           120          100.0   2013-06-24  0.200000
8 2015-12-31    C           100          130.0   2014-11-14 -0.230769