我不知道如何在没有四个嵌套的for
循环的情况下执行此操作。
我想将 自定义函数 应用于hour
和day
的每个可能的子集组合,返回该值,并然后将数据框旋转到方形矩阵中。但是,这些for
循环似乎过多,因此我正在寻找一种更有效的方法。我拥有的数据相当大,因此速度上的任何提高都是有益的。
编辑:我更新了问题,以包括自定义功能。
这里是一个例子,
样本数据
import pandas as pd
import numpy as np
dat = pd.DataFrame({'day': {0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 1, 10: 2, 11: 2, 12: 2, 13: 2, 14: 2, 15: 2, 16: 2, 17: 2, 18: 2, 19: 2}, 'hour': {0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 2, 6: 2, 7: 2, 8: 2, 9: 2, 10: 1, 11: 1, 12: 1, 13: 1, 14: 1, 15: 2, 16: 2, 17: 2, 18: 2, 19: 2}, 'distance': {0: 1.2898851269657656, 1: 0.0, 2: 0.8371526423804061, 3: 0.8703856587273138, 4: 0.6257425922449789, 5: 0.0, 6: 0.0, 7: 0.0, 8: 1.2895328696587023, 9: 0.0, 10: 0.6875527848294374, 11: 0.0, 12: 0.0, 13: 0.9009031833559706, 14: 0.0, 15: 1.1040652963428623, 16: 0.0, 17: 0.0, 18: 0.0, 19: 0.0}})
代码
def custom_fn(x, y):
x = pd.Series(x)
y = pd.Series(y)
x = x**2
y = np.sqrt(y)
return x.sum() - y.sum()
# Empty data.frame to append to
dmat = pd.DataFrame()
# For i, j = hour; k, l = day
for i in range(1, 3):
for j in range(1, 3):
for k in range(1, 3):
for l in range(1, 3):
x = dat[(dat['hour'] == i) & (dat['day'] == k)].distance
y = dat[(dat['hour'] == j) & (dat['day'] == l)].distance
# Calculate difference
jds = custom_fn(x, y)
# Build data frame and append
outdat = pd.DataFrame({'day_hour_a': f"{k}_{i}", 'day_hour_b': f"{l}_{j}", 'jds': [round(jds, 4)]})
dmat = dmat.append(outdat, ignore_index=True)
# Pivot data to get matrix
distMatrix = dmat.pivot(index='day_hour_a', columns='day_hour_b', values='jds')
输出
> print(distMatrix)
day_hour_b 1_1 1_2 2_1 2_2
day_hour_a
1_1 -0.2609 2.3782 1.7354 2.4630
1_2 -2.1118 0.5273 -0.1155 0.6121
2_1 -2.4903 0.1488 -0.4940 0.2336
2_2 -2.5557 0.0834 -0.5594 0.1682
答案 0 :(得分:2)
如果我理解正确,那么您正在做的事情与以下内容相同:
def f(x):
return x.mean()
x = df.groupby(['day', 'hour'])['distance'].apply(f)
x = x.values[:,None] - x.values
print(x)
输出:
[[ 0. 0.46672663 0.40694201 0.50382014]
[-0.46672663 0. -0.05978462 0.03709351]
[-0.40694201 0.05978462 0. 0.09687813]
[-0.50382014 -0.03709351 -0.09687813 0. ]]
更新:对于更新的自定义功能,您仍然可以将其分解为单独的groupby
:
g = df.groupby(['day', 'hour'])['distance']
x = g.apply(lambda z: (z**2).sum())
y = g.apply(lambda z: np.sqrt(z).sum())
x.values[:,None] - y.values
输出:
array([[-0.26092193, 2.37817717, 1.73540595, 2.46300806],
[-2.11178008, 0.52731901, -0.1154522 , 0.61214991],
[-2.49031973, 0.14877937, -0.49399185, 0.23361026],
[-2.55571493, 0.08338417, -0.55938705, 0.16821506]])
更新2::如果无法分开计算,则另一种选择是:
def f(x, y):
return distance.jensenshannon(x, y)
x = []
g = df.groupby(['day', 'hour'])['distance']
for k1, g1 in g:
for k2, g2 in g:
x += [(k1, k2, f(g1, g2))]
x = pd.DataFrame(x).pivot(index=0, columns=1, values=2)
print(x)
输出:
1 (1, 1) (1, 2) (2, 1) (2, 2)
0
(1, 1) 0.000000 0.623167 0.419371 0.550291
(1, 2) 0.623167 0.000000 0.424608 0.832555
(2, 1) 0.419371 0.424608 0.000000 0.504233
(2, 2) 0.550291 0.832555 0.504233 0.000000