joblib:并行始终陷入困境

时间:2017-04-24 10:54:50

标签: python parallel-processing joblib

我的并行使用:

sdf = Parallel(n_jobs=3, verbose=1, pre_dispatch='1.5*n_jobs')(delayed(char_etl)(x,k1,k2,k3) for x in X)

X是一个列表.char_etl是一个sring匹配函数。 版本:python2.7.11,centos6.7,joblib 0.11 每次都被卡住了,就像这样

[Parallel(n_jobs=3)]: Done 89 tasks | elapsed: 10.9s
[Parallel(n_jobs=3)]: Done 258 tasks | elapsed: 46.8s
[Parallel(n_jobs=3)]: Done 552 tasks | elapsed: 1.7min
[Parallel(n_jobs=3)]: Done 902 tasks | elapsed: 2.7min
[Parallel(n_jobs=3)]: Done 997 out of 1000 | elapsed: 2.9min remaining: 0.5s
Thanks.

0 个答案:

没有答案