我可以在项目中使用更多帮助。我试图分析450万行数据。我已将数据读入数据框,已组织数据,现在有3列:1)日期为datetime 2)唯一标识符3)价格
我需要计算每件商品价格的年度变化,但日期不统一且每件商品不一致。例如:
date item price
12/31/15 A 110
12/31/15 B 120
12/31/14 A 100
6/24/13 B 100
我希望找到的结果是:
date item price previousdate % change
12/31/15 A 110 12/31/14 10%
12/31/15 B 120 6/24/13 20%
12/31/14 A 100
6/24/13 B 100
编辑 - 更好的数据示例
date item price
6/1/2016 A 276.3457646
6/1/2016 B 5.044165645
4/27/2016 B 4.91300186
4/27/2016 A 276.4329163
4/20/2016 A 276.9991265
4/20/2016 B 4.801263717
4/13/2016 A 276.1950213
4/13/2016 B 5.582923328
4/6/2016 B 5.017863509
4/6/2016 A 276.218649
3/30/2016 B 4.64274783
3/30/2016 A 276.554653
3/23/2016 B 5.576438253
3/23/2016 A 276.3135836
3/16/2016 B 5.394435443
3/16/2016 A 276.4222986
3/9/2016 A 276.8929462
3/9/2016 B 4.999951262
3/2/2016 B 4.731349423
3/2/2016 A 276.3972068
1/27/2016 A 276.8458971
1/27/2016 B 4.993033132
1/20/2016 B 5.250379701
1/20/2016 A 276.2899864
1/13/2016 B 5.146639666
1/13/2016 A 276.7041978
1/6/2016 B 5.328296958
1/6/2016 A 276.9465891
12/30/2015 B 5.312301356
12/30/2015 A 256.259668
12/23/2015 B 5.279105491
12/23/2015 A 255.8411198
12/16/2015 B 5.150798234
12/16/2015 A 255.8360529
12/9/2015 A 255.4915183
12/9/2015 B 4.722876886
12/2/2015 A 256.267146
12/2/2015 B 5.083626167
10/28/2015 B 4.876177757
10/28/2015 A 255.6464653
10/21/2015 B 4.551439655
10/21/2015 A 256.1735769
10/14/2015 A 255.9752668
10/14/2015 B 4.693967392
10/7/2015 B 4.911797443
10/7/2015 A 256.2556707
9/30/2015 B 4.262994526
9/30/2015 A 255.8068691
7/1/2015 A 255.7312385
4/22/2015 A 234.6210132
4/15/2015 A 235.3902076
4/15/2015 B 4.154926102
4/1/2015 A 234.4713827
2/25/2015 A 235.1391496
2/18/2015 A 235.1223471
我所做的事情(在其他用户的帮助下)并没有起作用,但却在下面。感谢您提供的任何帮助,或指出我正确的方向!
import pandas as pd
import datetime as dt
import numpy as np
df = pd.read_csv('...python test file5.csv',parse_dates =['As of Date'])
df = df[['item','price','As of Date']]
def get_prev_year_price(x, df):
try:
return df.loc[x['prev_year_date'], 'price']
#return np.abs(df.time - x)
except Exception as e:
return x['price']
#Function to determine the closest date from given date and list of all dates
def nearest(items, pivot):
return min(items, key=lambda x: abs(x - pivot))
df['As of Date'] = pd.to_datetime(df['As of Date'],format='%m/%d/%Y')
df = df.rename(columns = {df.columns[2]:'date'})
# list of dates
dtlst = [item for item in df['date']]
data = []
data2 = []
for item in df['item'].unique():
item_df = df[df['item'] == item] #select based on items
select_dates = item_df['date'].unique()
item_df.set_index('date', inplace=True) #set date as key index
item_df = item_df.resample('D').mean().reset_index() #fill in missing date
item_df['price'] = item_df['price'].interpolate('nearest') #fill in price with nearest price available
# use max(item_df['date'] where item_df['date'] < item_df['date'] - pd.DateOffset(years=1, days=1))
#possible_date = item_df['date'] - pd.DateOffset(years=1)
#item_df['prev_year_date'] = max(df[df['date'] <= possible_date])
item_df['prev_year_date'] = item_df['date'] - pd.DateOffset(years=1) #calculate 1 year ago date
date_df = item_df[item_df.date.isin(select_dates)] #select dates with useful data
item_df.set_index('date', inplace=True)
date_df['prev_year_price'] = date_df.apply(lambda x: get_prev_year_price(x, item_df),axis=1)
#date_df['prev_year_price'] = date_df.apply(lambda x: nearest(dtlst, x),axis=1)
date_df['change'] = date_df['price'] / date_df['prev_year_price']-1
date_df['item'] = item
data.append(date_df)
data2.append(item_df)
summary = pd.concat(data).sort_values('date', ascending=False)
#print (summary)
#saving the output of the CSV file to see how data looks after being handled
filename = '...python_test_file_save4.csv'
summary.to_csv(filename, index=True, encoding='utf-8')
答案 0 :(得分:2)
使用当前的用例假设,这适用于此特定用例
<p-checkbox [value]="item1" formControlName="selectedComponents" [label]="item1.itemSku"></p-checkbox>
排序In [2459]: def change(grp):
...: grp['% change'] = grp.price.diff()
...: grp['previousdate'] = grp.date.shift(1)
...: return grp
然后date
和groupby
apply
函数,然后对索引进行排序。
change
答案 1 :(得分:1)
merge_asof
这是一个很好的情况,它通过查找右数据帧的最后一行(小于左数据帧的键)来合并两个数据帧。我们需要首先在正确的数据框架中添加一年,因为日期之间的要求是1年或更长时间。
以下是您在评论中提到的一些示例数据。
date item price
12/31/15 A 110
12/31/15 B 120
12/31/14 A 100
6/24/13 B 100
12/31/15 C 100
1/31/15 C 80
11/14/14 C 130
11/19/13 C 110
11/14/13 C 200
日期需要按merge_asof
排序才能生效。 merge_asof
也会删除加入列,因此我们需要将其副本放回正确的数据框中。
设置数据框
df = df.sort_values('date')
df_copy = df.copy()
df_copy['previousdate'] = df_copy['date']
df_copy['date'] += pd.DateOffset(years=1)
使用merge_asof
df_final = pd.merge_asof(df, df_copy,
on='date',
by='item',
suffixes=['current', 'previous'])
df_final['% change'] = (df_final['pricecurrent'] - df_final['priceprevious']) / df_final['priceprevious']
df_final
date item pricecurrent priceprevious previousdate % change
0 2013-06-24 B 100 NaN NaT NaN
1 2013-11-14 C 200 NaN NaT NaN
2 2013-11-19 C 110 NaN NaT NaN
3 2014-11-14 C 130 200.0 2013-11-14 -0.350000
4 2014-12-31 A 100 NaN NaT NaN
5 2015-01-31 C 80 110.0 2013-11-19 -0.272727
6 2015-12-31 A 110 100.0 2014-12-31 0.100000
7 2015-12-31 B 120 100.0 2013-06-24 0.200000
8 2015-12-31 C 100 130.0 2014-11-14 -0.230769