Question

对于财务应用程序，我正在尝试创建一个DataFrame，其中每一行都是特定权益的会话日期值。要获取数据，我正在使用Pandas Remote Data。因此，例如，我正在尝试创建的功能可能是之前32个会话的调整后的结束。

这在for循环中很容易实现，但是对于大型功能集需要相当长的时间（例如，在“ge”上返回1960并使每行包含前面的256个会话值）。有没有人看到一个很好的方法来矢量化这段代码？

import pandas as pd

def featurize(equity_data, n_sessions, col_label='Adj Close'):
    """ 
    Generate a raw (unnormalized) feature set from the input data.
    The value at col_label on the given date is taken
    as a feature, and each row contains values for n_sessions
    """
    features = pd.DataFrame(index=equity_data.index[(n_sessions - 1):],
            columns=range((-n_sessions + 1), 1)) 
    for i in range(len(features.index)):
        features.iloc[i, :] = equity_data[i:(n_sessions + i)][col_label].values
    return features

我也可以轻松地多线程，但我猜测，如果我可以对它进行矢量化，那么大熊猫会自动执行此操作。我提到这主要是因为我主要关心的是表现。因此，如果多线程可能以任何显着的方式超越矢量化，那么我宁愿这样做。

输入和输出的简短示例：

>>> eq_data Open High Low Close Volume Adj Close Date 2014-01-02 15.42 15.45 15.28 15.44 31528500 14.96 2014-01-03 15.52 15.64 15.30 15.51 46122300 15.02 2014-01-06 15.72 15.76 15.52 15.58 42657600 15.09 2014-01-07 15.73 15.74 15.35 15.38 54476300 14.90 2014-01-08 15.60 15.71 15.51 15.54 48448300 15.05 2014-01-09 15.83 16.02 15.77 15.84 67836500 15.34 2014-01-10 16.01 16.11 15.94 16.07 44984000 15.57 2014-01-13 16.37 16.53 16.08 16.11 57566400 15.61 2014-01-14 16.31 16.43 16.17 16.40 44039200 15.89 2014-01-15 16.37 16.73 16.35 16.70 64118200 16.18 2014-01-16 16.67 16.76 16.56 16.73 38410800 16.21 2014-01-17 16.78 16.78 16.45 16.52 37152100 16.00 2014-01-21 16.64 16.68 16.36 16.41 35597200 15.90 2014-01-22 16.44 16.62 16.37 16.55 28741900 16.03 2014-01-23 16.49 16.53 16.31 16.43 37860800 15.92 2014-01-24 16.19 16.21 15.78 15.83 66023500 15.33 2014-01-27 15.90 15.91 15.52 15.71 51218700 15.22 2014-01-28 15.97 16.01 15.51 15.72 57677500 15.23 2014-01-29 15.48 15.53 15.20 15.26 52241500 14.90 2014-01-30 15.43 15.45 15.18 15.25 32654100 14.89 2014-01-31 15.09 15.10 14.90 14.96 64132600 14.61 >>> features = data.featurize(eq_data, 3) >>> features -2 -1 0 Date 2014-01-06 14.96 15.02 15.09 2014-01-07 15.02 15.09 14.9 2014-01-08 15.09 14.9 15.05 2014-01-09 14.9 15.05 15.34 2014-01-10 15.05 15.34 15.57 2014-01-13 15.34 15.57 15.61 2014-01-14 15.57 15.61 15.89 2014-01-15 15.61 15.89 16.18 2014-01-16 15.89 16.18 16.21 2014-01-17 16.18 16.21 16 2014-01-21 16.21 16 15.9 2014-01-22 16 15.9 16.03 2014-01-23 15.9 16.03 15.92 2014-01-24 16.03 15.92 15.33 2014-01-27 15.92 15.33 15.22 2014-01-28 15.33 15.22 15.23 2014-01-29 15.22 15.23 14.9 2014-01-30 15.23 14.9 14.89 2014-01-31 14.9 14.89 14.61

因此，每行要素都是n_sessions DataFrame的“Adj Close”列中的一系列3（features）个连续值。

====================

改进版基于Primer的答案：

def featurize(equity_data, n_sessions, column='Adj Close'): """ Generate a raw (unnormalized) feature set from the input data. The value at column on the given date is taken as a feature, and each row contains values for n_sessions >>> timeit.timeit('data.featurize(data.get("ge", dt.date(1960, 1, 1), dt.date(2014, 12, 31)), 256)', setup=s, number=1) 1.6771750450134277 """ features = pd.DataFrame(index=equity_data.index[(n_sessions - 1):], columns=map(str, range((-n_sessions + 1), 1)), dtype='float64') values = equity_data[column].values for i in range(n_sessions - 1): features.iloc[:, i] = values[i:(-n_sessions + i + 1)] features.iloc[:, n_sessions - 1] = values[(n_sessions - 1):] return features

Answer 1

看起来shift是你的朋友，这样的事情会发生：

df = pd.DataFrame({'adj close': np.random.random(10) + 15},index=pd.date_range(start='2014-01-02', periods=10, freq='B'))
df.index.name = 'date'
df

                adj close
date                 
2014-01-02     15.650
2014-01-03     15.775
2014-01-06     15.750
2014-01-07     15.464
2014-01-08     15.966
2014-01-09     15.475
2014-01-10     15.164
2014-01-13     15.281
2014-01-14     15.568
2014-01-15     15.648

features = pd.DataFrame(data=df['adj close'], index=df.index)
features.columns = ['0']
features['-1'] = df['adj close'].shift()
features['-2'] = df['adj close'].shift(2)
features.dropna(inplace=True)
features

                 0      -1      -2
date                              
2014-01-06  15.750  15.775  15.650
2014-01-07  15.464  15.750  15.775
2014-01-08  15.966  15.464  15.750
2014-01-09  15.475  15.966  15.464
2014-01-10  15.164  15.475  15.966
2014-01-13  15.281  15.164  15.475
2014-01-14  15.568  15.281  15.164
2014-01-15  15.648  15.568  15.281

vectorize for-loop填充Pandas DataFrame

1 个答案: