我在pandas python DataFrame中有一个大的相关矩阵:df(342,342)。
如何获取上三角形中所有数字的均值,sd等,不包括沿对角线的1?
谢谢。
答案 0 :(得分:5)
另一个潜在的答案是:
In [1]: corr
Out[1]:
a b c d e
a 1.000000 0.022246 0.018614 0.022592 0.008520
b 0.022246 1.000000 0.033029 0.049714 -0.008243
c 0.018614 0.033029 1.000000 -0.016244 0.049010
d 0.022592 0.049714 -0.016244 1.000000 -0.015428
e 0.008520 -0.008243 0.049010 -0.015428 1.000000
In [2]: corr.values[np.triu_indices_from(corr.values,1)].mean()
Out[2]: 0.016381
修改:添加了效果指标
我的解决方案的表现:
In [3]: %timeit corr.values[np.triu_indices_from(corr.values,1)].mean()
10000 loops, best of 3: 48.1 us per loop
Theodros Zelleke的单线解决方案的表现:
In [4]: %timeit corr.unstack().ix[zip(*np.triu_indices_from(corr, 1))].mean()
1000 loops, best of 3: 823 us per loop
DSM解决方案的表现:
In [5]: def method1(df):
...: df2 = df.copy()
...: df2.values[np.tril_indices_from(df2)] = np.nan
...: return df2.unstack().mean()
...:
In [5]: %timeit method1(corr)
1000 loops, best of 3: 242 us per loop
答案 1 :(得分:3)
这很有趣。我不保证这是真正的大熊猫;我仍然处于学习pandas
的“numpy +更好的索引”阶段。也就是说,这样的事情应该可以完成工作。
首先,我们制作玩具相关矩阵:
>>> import pandas as pd
>>> import numpy as np
>>> frame = pd.DataFrame(np.random.randn(1000, 5), columns=['a', 'b', 'c', 'd', 'e'])
>>> corr = frame.corr()
>>> corr
a b c d e
a 1.000000 0.022246 0.018614 0.022592 0.008520
b 0.022246 1.000000 0.033029 0.049714 -0.008243
c 0.018614 0.033029 1.000000 -0.016244 0.049010
d 0.022592 0.049714 -0.016244 1.000000 -0.015428
e 0.008520 -0.008243 0.049010 -0.015428 1.000000
然后我们制作副本,并使用tril_indices_from获取较低的索引来掩盖它们:
>>> c2 = corr.copy()
>>> c2.values[np.tril_indices_from(c2)] = np.nan
>>> c2
a b c d e
a NaN 0.06952 -0.021632 -0.028412 -0.029729
b NaN NaN -0.022343 -0.063658 0.055247
c NaN NaN NaN -0.013272 0.029102
d NaN NaN NaN NaN -0.046877
e NaN NaN NaN NaN NaN
现在我们可以对展平数组进行统计:
>>> c2.unstack().mean()
-0.0072054178481488901
>>> c2.unstack().std()
0.043839624201635466