running joblib.Parallel(mlxtend) does not scale in cloud-ml

Question

Im running a job using the mlxtend library. Specifically the sequential_feature_selector that is parallelized using joblib.Parallel source. When I run the package on my local computer it uses all the available CPUs, but when i send the job to cloud-ml it only uses one core. It doesn't matter what is the number that i put in the n_jobs parameter. I´ve also tried with differents machine types but same thing happen. Does anybody know what the problem might be ?

Answer 1

对于任何可能感兴趣的人，我们解决了将sklearn中的setup.py版本固定为0.20.2的问题。我们以前在软件包中有sklearn，但没有版本。

#setup.py
from setuptools import find_packages
from setuptools import setup

REQUIRED_PACKAGES = ['joblib==0.13.0',
                     'scikit-learn==0.20.2',
                     'mlxtend']