Question

我想在时间索引的pandas DataFrame中添加列，其中包含每个现有列在过去n小时内的更改率。我用以下代码完成了这个，但是，它对我的需求来说太慢了（可能是因为循环遍历每列的每个索引？）。

是否有（计算上）更快的方法来做到这一点？

roc_hours = 12
tol = 1e-10 
for c in ts.columns:
    c_roc = c + ' +++ RoC ' + str(roc_hours) + 'h' 
    ts[c_roc] = np.nan
    for i in ts.index[np.isfinite(ts[c])]:
        df = ts[c][i - np.timedelta64(roc_hours, 'h'):i]
        X = (df.index.values - df.index.values.min()).astype('Int64')*2.77778e-13 #hours back
        Y = df.values
        if Y.std() > tol and X.shape[0] > 1:
            fit = np.polyfit(X,Y,1)
            ts[c_roc][i] = fit[0]
        else:
            ts[c_roc][i] = 0

编辑输入数据帧ts是不规则采样的，可以包含NaN个。前几行输入ts：

+---------------------+-------------------+------+------+--------------------+-------------------+------------------+
|         WCT         |         a         |  b   |  c   |         d          |         e         |        f         |
+---------------------+-------------------+------+------+--------------------+-------------------+------------------+
| 2011-09-04 20:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-04 21:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-04 22:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-04 23:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-05 02:00:00 |        93.0       | 97.0 | 20.0 |       209.0        |        85.0       |       98.0       |
| 2011-09-05 03:00:00 | 74.14285714285714 | 97.0 | 20.0 | 194.14285714285717 | 74.42857142857143 |       98.0       |
| 2011-09-05 04:00:00 |        67.5       | 98.5 | 20.0 |       176.0        |        75.0       |       98.0       |
| 2011-09-05 05:00:00 |        72.0       | 98.5 | 20.0 |       176.0        |        75.0       |       98.0       |
| 2011-09-05 07:00:00 |        80.0       | 93.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 08:00:00 |        80.0       | 93.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 09:00:00 |        78.5       | 98.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 10:00:00 |        73.0       | 98.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 11:00:00 |        77.0       | 98.0 | 18.0 |       175.0        |        87.0       | 97.0999984741211 |
| 2011-09-05 12:00:00 |        78.0       | 98.0 | 19.0 |       163.0        |        57.0       | 98.4000015258789 |
| 2011-09-05 15:00:00 |        78.0       | 98.0 | 19.0 |       163.0        |        57.0       | 98.4000015258789 |
+---------------------+-------------------+------+------+--------------------+-------------------+------------------+

修改2

分析后，瓶颈在于切片步骤：df = ts[c][i - np.timedelta64(roc_hours, 'h'):i]。这条线拉出了now-roc_hours和现在之间的时间戳记。这是非常方便的语法，但占用了大部分的计算时间。

Answer 1

适用于我的数据集，但未经过检查：

import pandas as pd
from numpy import polyfit
from matplotlib import style
style.use('ggplot')

# ... acquire a dataframe named *water* with a column *value*

WINDOW = 10
ax=water.value.plot()
roll = pd.rolling_mean(water.value, WINDOW)
roll.plot(ax=ax)

def lintrend(df):
    df = df.tolist()
    m, b = polyfit(range(len(df)), df,1)
    return m

linny = pd.rolling_apply(water.value, WINDOW, lintrend)

linny.plot(ax=ax)

将rolling_apply转换为numpy.ndarray后，将numpy.ndarray转换为列表似乎不够优雅。建议？

使用pandas时间序列在过去n小时内的变化率

1 个答案: