我已经开始使用scikit-garden
软件包中的分位数随机森林(QRF)。以前,我是使用RandomForestRegresser
中的sklearn.ensemble
创建常规随机森林的。
在数据集较小的情况下,QRF的速度似乎与常规RF相当,但是随着数据大小的增加,QRF的预测速度要比RF慢得多。
这是预期的吗?如果是这样,有人可以解释一下为什么要花这么长时间才能做出这些预测和/或就如何更及时地获得分位数预测给出任何建议。
有关玩具示例,请参见下文,我在其中测试了各种数据集大小的训练时间和预测时间。
import matplotlib as mpl
mpl.use('Agg')
from sklearn.ensemble import RandomForestRegressor
from skgarden import RandomForestQuantileRegressor
from sklearn.model_selection import train_test_split
import numpy as np
import time
import matplotlib.pyplot as plt
log_ns = np.arange(0.5, 5, 0.5) # number of observations (log10)
ns = (10 ** (log_ns)).astype(int)
print(ns)
m = 14 # number of covariates
train_rf = []
train_qrf = []
pred_rf = []
pred_qrf = []
for n in ns:
# create dataset
print('n = {}'.format(n))
print('m = {}'.format(m))
rndms = np.random.normal(size=n)
X = np.random.uniform(size=[n,m])
betas = np.random.uniform(size=m)
y = 3 + np.sum(betas[None,:] * X, axis=1) + rndms
# split test/train
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
# random forest
rf = RandomForestRegressor(n_estimators=1000, random_state=0)
st = time.time()
rf.fit(X_train, y_train)
en = time.time()
print('Fit time RF = {} secs'.format(en - st))
train_rf.append(en - st)
# quantile random forest
qrf = RandomForestQuantileRegressor(random_state=0, min_samples_split=10, n_estimators=1000)
qrf.set_params(max_features = X.shape[1] // 3)
st = time.time()
qrf.fit(X_train, y_train)
en = time.time()
print('Fit time QRF = {} secs'.format(en - st))
train_qrf.append(en - st)
# predictions
st = time.time()
preds_rf = rf.predict(X_test)
en = time.time()
print('Prediction time RF = {}'.format(en - st))
pred_rf.append(en - st)
st = time.time()
preds_qrf = qrf.predict(X_test, quantile=50)
en = time.time()
print('Prediction time QRF = {}'.format(en - st))
pred_qrf.append(en - st)
fig, ax = plt.subplots()
ax.plot(np.log10(ns), train_rf, label='RF train', color='blue')
ax.plot(np.log10(ns), train_qrf, label='QRF train', color='red')
ax.plot(np.log10(ns), pred_rf, label='RF predict', color='blue', linestyle=':')
ax.plot(np.log10(ns), pred_qrf, label='QRF predict', color='red', linestyle =':')
ax.legend()
ax.set_xlabel('log(n)')
ax.set_ylabel('time (s)')
fig.savefig('time_comparison.png')
这是输出: Time comparison of RF and QRF training and predictions
答案 0 :(得分:2)
我不是这个或任何分位数回归软件包的开发人员,但是我查看了scikit-garden和quantRegForest / ranger的源代码,并且我对为什么R版本这么快的原因有所了解:
编辑:On a related github issue,lmssdd提到了该方法的效果远比论文中的“标准程序”差。我尚未详细阅读该论文,因此请对此表示怀疑。
skgarden predict
函数的基本思想是保存与所有叶子对应的所有y_train
值。然后,在预测新样本时,您将收集相关的叶子和相应的y_train
值,并计算该数组的(加权)分位数。 R版本采用了一种快捷方式:每个叶子节点仅保存一个随机选择的y_train
值。这有两个优点:由于每个叶节点中始终只有一个值,因此使相关y_train
值的收集变得简单得多。其次,由于每片叶子的权重都完全相同,因此分位数的计算要简单得多。
由于每个叶子只使用一个(随机)值,而不是所有叶子,因此这是一种近似方法。以我的经验,如果您有足够的树木(至少50至100棵左右),则对结果的影响很小。但是,我对数学的了解还不足以说近似值到底有多好。
以下是针对RandomForestQuantileRegressor模型的分位数预测的更简单R方法的实现。请注意,函数的前半部分是(一次性)选择每片叶子的y_train值的过程。如果作者要在skgarden中实现此方法,他们将把这部分逻辑上移至fit
方法,仅保留最后6行左右,这将使predict
方法更快。同样在我的示例中,我使用的分位数是0到1,而不是0到100。
def predict_approx(model, X_test, quantiles=[0.05, 0.5, 0.95]):
"""
Function to predict quantiles much faster than the default skgarden method
This is the same method that the ranger and quantRegForest packages in R use
Output is (n_samples, n_quantiles) or (n_samples, ) if a scalar is given as quantiles
"""
# Begin one-time calculation of random_values. This only depends on model, so could be saved.
n_leaves = np.max(model.y_train_leaves_) + 1 # leaves run from 0 to max(leaf_number)
random_values = np.zeros((model.n_estimators, n_leaves))
for tree in range(model.n_estimators):
for leaf in range(n_leaves):
train_samples = np.argwhere(model.y_train_leaves_[tree, :] == leaf).reshape(-1)
if len(train_samples) == 0:
random_values[tree, leaf] = np.nan
else:
train_values = model.y_train_[train_samples]
random_values[tree, leaf] = np.random.choice(train_values)
# Optionally, save random_values as a model attribute for reuse later
# For each sample, get the random leaf values from all the leaves they land in
X_leaves = model.apply(X_test)
leaf_values = np.zeros((X_test.shape[0], model.n_estimators))
for i in range(model.n_estimators):
leaf_values[:, i] = random_values[i, X_leaves[:, i]]
# For each sample, calculate the quantiles of the leaf_values
return np.quantile(leaf_values, np.array(quantiles), axis=1).transpose()