与`map_partitions`并行化预测

Question

我有一个形状为（25M，79）的数据框，正在尝试并行化sklearn管道预测。

当我仅对一个分区运行它时，它会按预期工作：

n_partitions = 1000
ddf = dd.from_pandas(df_x_selection, npartitions=n_partitions)
grid_searcher.best_estimator_.predict_proba(ddf.get_partition(0))

但是如果我将其应用于每个分区，则它将失败：

n_partitions = 1000
ddf = dd.from_pandas(df_x_selection, npartitions=n_partitions)

def _f(_df, _pipeline, _predicted_class) -> np.array:
    return _pipeline.predict_proba(_df)[:, _predicted_class]

ddf.map_partitions(_f, grid_searcher.best_estimator_, 1, meta=(None, 'f8')).compute()

错误是：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/pandas/core/internals/blocks.py in __init__(self, values, placement, ndim)
    130             raise ValueError(
--> 131                 f"Wrong number of items passed {len(self.values)}, "
    132                 f"placement implies {len(self.mgr_locs)}"

ValueError: Wrong number of items passed 79, placement implies 100

我在做什么错？ 谢谢