Question

我创建了一个类似于sklearn gridsearch的小代码，它在一组超参数上训练模型（在下面的代码中的X和y上），使用几个指标检查模型性能验证数据（ Xt，yt_class）并将结果存储在pandas DataFrame中。

    from sklearn.grid_search import ParameterGrid
    from sklearn.metrics import precision_score,f1_score

    grid = {'C':[1,10.0,50,100.0],'gamma':[0.00001,0.0001,0.001,0.01,0.1]}
    param_grid = ParameterGrid(grid)
    results = pd.DataFrame(list(param_grid))
    precision = []
    f1 = []
    for params in param_grid:
        model = SVC(kernel='rbf',cache_size=1000,class_weight='balanced',**params)
        model.fit(X,y)
        precision.append(precision_score(yt_class, model.predict(Xt), average='weighted'))
        f1.append(f1_score(yt_class, model.predict(Xt), average='weighted'))
        print(params)
        print(precision_score(yt_class, model.predict(Xt), average='weighted'))
        print(f1_score(yt_class, model.predict(Xt), average='weighted'))

    results['precision'] = precision
    results['f1'] = f1

现在我试图让我的循环在多个CPU上运行，我尝试按照多处理模块的基本示例进行操作，但是对Python和程序设计的新手并不能解决它在我的情况下的作用。

不起作用的示例：

import multiprocessing as mp
pool = mp.Pool(processes=8)

def get_scores(param_grid):
    precision = []
    f1 = []
    for params in param_grid:
        model = SVC(kernel='rbf',cache_size=1000,class_weight='balanced',**params)
        model.fit(X,y)
        model.predict(Xt)
        precision.append(precision_score(yt_class, model.predict(Xt), average='weighted'))
        f1.append(f1_score(yt_class, model.predict(Xt), average='weighted'))
    return precision,f1    

scores = pool.apply(get_scores,param_grid)

Answer 1

您的get_scores方法应该只包含循环的内部部分

试试这个：

import multiprocessing as mp
from sklearn.grid_search import ParameterGrid
from sklearn.metrics import precision_score,f1_score

def get_scores(params):
    model = SVC(kernel='rbf',cache_size=1000,class_weight='balanced',**params)
    model.fit(X,y)
    model.predict(Xt)
    precision = precision_score(yt_class, model.predict(Xt), average='weighted')
    f1 = f1_score(yt_class, model.predict(Xt), average='weighted')
    return precision, f1    


grid = {'C':[1,10.0,50,100.0],'gamma':[0.00001,0.0001,0.001,0.01,0.1]}
param_grid = ParameterGrid(grid)
pool = mp.Pool(processes=8)

scores = pool.map_async(get_scores, param_grid).get()
# scores is a list of tuples [(precision_1, f1_1), (precision_2, f1_2)...]
# you can "unzip" it like this

precision, f1 = zip(*scores)

Python，在几个CPU上运行循环

1 个答案: