我正在尝试将dask
和fbprophet
库一起使用,我要么做错了,要么出现意外的性能问题。
import dask.dataframe as dd
import datetime as dt
import multiprocessing as mp
import numpy as np
import pandas as pd
pd.options.mode.chained_assignment = None
from fbprophet import Prophet
import time
ncpu = mp.cpu_count()
def parallel_pd(fun, vec, pool = ncpu-1):
with mp.Pool(pool) as p:
res = p.map(fun,vec)
return(res)
def forecast1dd(ts):
time.sleep(0.1)
return ts["y"].max()
def forecast1mp(key):
ts = df[df["key"]==key]
time.sleep(0.1)
return ts["y"].max()
def forecast2dd(ts):
future = pd.DataFrame({"ds":pd.date_range(start=ts["ds"].max()+ dt.timedelta(days=1),
periods=7, freq="D")})
key = ts.name
model = Prophet(yearly_seasonality=True)
model.fit(ts)
forecast = model.predict(future)
future["yhat"] = forecast["yhat"]
future["key"] = key
return future.as_matrix()
def forecast2mp(key):
ts = df[df["key"]==key]
future = pd.DataFrame({"ds":pd.date_range(start=ts["ds"].max()+ dt.timedelta(days=1),
periods=7, freq="D")})
model = Prophet(yearly_seasonality=True)
model.fit(ts)
forecast = model.predict(future)
future["yhat"] = forecast["yhat"]
future["key"] = key
return future.as_matrix()
在一方面,我有一个自定义函数,运行时间约为0.1秒,因此forecast1dd
和forecast1mp
模拟我的函数和以下数据框
N = 2*365
key_n = 5000
df = pd.concat([pd.DataFrame({"ds":pd.date_range(start="2015-01-01",periods=N, freq="D"),
"y":np.random.normal(100,20,N),
"key":np.repeat(str(k),N)}) for k in range(key_n)])
keys = df.key.unique()
df = df.sample(frac=1).reset_index(drop=True)
ddf = dd.from_pandas(df, npartitions=ncpu*2)
我(分别)获得
%%time
grp = ddf.groupby("key").apply(forecast1dd, meta=pd.Series(name="s"))
df1dd = grp.to_frame().compute()
CPU times: user 7.7 s, sys: 400 ms, total: 8.1 s
Wall time: 1min 8s
%%time
res = parallel_pd(forecast1mp,keys)
CPU times: user 820 ms, sys: 360 ms, total: 1.18 s
Wall time: 10min 36s
在第一种情况下,核心不是100%使用,但性能符合我的实际情况。使用行分析器很容易检查第二种情况下性能降低的罪魁祸首是ts = df[df["key"]==key]
,如果我们有更多密钥,事情会变得更糟。
到目前为止,我对dask
感到满意。但每当我尝试使用fbprophet
时,事情都会发生变化。在这里,我使用的keys
更少,但前一个案例dask
的效果总是不如multiprocessing
。
N = 2*365
key_n = 200
df = pd.concat([pd.DataFrame({"ds":pd.date_range(start="2015-01-01",periods=N, freq="D"),
"y":np.random.normal(100,20,N),
"key":np.repeat(str(k),N)}) for k in range(key_n)])
keys = df.key.unique()
df = df.sample(frac=1).reset_index(drop=True)
ddf = dd.from_pandas(df, npartitions=ncpu*2)
%%time
grp = ddf.groupby("key").apply(forecast2dd,
meta=pd.Series(name="s")).to_frame().compute()
df2dd = pd.concat([pd.DataFrame(a) for a in grp.s.values])
CPU times: user 3min 42s, sys: 15 s, total: 3min 57s
Wall time: 3min 30s
%%time
res = parallel_pd(forecast2mp,keys)
df2mp = pd.concat([pd.DataFrame(a) for a in res])
CPU times: user 76 ms, sys: 160 ms, total: 236 ms
Wall time: 39.4 s
现在我的问题是:
答案 0 :(得分:2)
我怀疑Prophet持有GIL,所以在计算ddf.groupby("key").apply(forecast2dd, meta=pd.Series(name="s")
时,只有一个线程可以同时运行Python代码。使用multiprocessing
可以避免这种情况,代价是必须复制数据ncpu
次。这应该与parallel_pd
函数具有相似的运行时间。
%%time
with dask.set_options(get=dask.multiprocessing.get):
grp = ddf.groupby("key").apply(forecast2dd,
meta=pd.Series(name="s")).to_frame().compute()
df2dd = pd.concat([pd.DataFrame(a) for a in grp.s.values])
CPU times: user 2.47 s, sys: 251 ms, total: 2.72 s
Wall time: 1min 27s
你可以尝试询问先知开发者是否需要持有GIL。我怀疑问题是在PyStan中,并且当实际的Stan解算器运行时他们可能不需要GIL。有一个Github问题here
附注:由于您的示例forecast1dd
是一个聚合,因此使用dd.Aggregation
可以更快地运行它:
%%time
def forcast1dd_chunk(ts):
time.sleep(0.1)
return ts.max()
def forecast1dd_agg(ts):
return ts.max()
f1dd = dd.Aggregation("forecast1dd", forcast1dd_chunk, forecast1dd_agg)
grp = ddf.groupby("key")[['y']].agg(f1dd)
x = grp.compute()
CPU times: user 59.5 ms, sys: 5.13 ms, total: 64.7 ms
Wall time: 355 ms
虽然这不符合您的实际问题,但这不是聚合。