Question

我正在使用多处理python库在机器学习问题中并行运行特征选择。此函数接受熊猫数据框作为输入并返回一些数字。

当我使用mp.pool.map()执行此功能时，一切运行顺利。但是，如果我将其替换为mp.pool.ThreadPool.map()，它将失败，并显示以下错误：

AssertionError: Number of manager items must equal union of block items # manager items: 15, # tot_items: 20。

奇怪的是，我一直在运行ThreadPool代码，直到昨天。然后，我尝试重新运行它并开始遇到这些错误。我需要ThreadPool，因为这是与IO绑定的工作，并且与pool相比，它的运行速度要快得多。

编辑：代码是这样的（python 2.7）：

import multiprocessing as mp
import pandas as pd (version 0.22.0)

def main_functionality(df, params):
    df = df[params['feature']]
    #Run 5-fold cross-validation
        data_df = pd.DataFrame(....)
        pred_df = pred_df.append(data_df)
    return statistics from pred_df

def a_function(df_init, feature, params_init):

    params = dict(params_init)
    df = df_init.copy()

    params['feature'] = feature
    try:
        results = main_functionality(df, params)
    except:
        results = (0,0,0)

    return results

def b_function(df, features):
    pool = mp.pool.ThreadPool(4)
    params = {...}
    results = pool.map(a_function,(df, feature, params) for f in features))

    results_df = pd.DataFrame(results)
    results_df.to_csv(...)

if __name__ == '__main__':
    df = read.csv(...) # A big CSV file (i.e. few GBs)
    features = [i for i in df.columns if i ....]

    b_function(df, features)

当mp.pool正常工作时，mp.pool.ThreadPool失败

0 个答案: