Question

我正在对虚拟数据帧train_final[categorical_var]进行虚拟编码。但是，当我运行代码时，出现内存错误。因为dask应该通过逐块加载数据来做到这一点，所以会发生这种情况。

代码如下：


from dask_ml.preprocessing import DummyEncoder
de = DummyEncoder()
train_final_cat = de.fit_transform(train_final[categorical_var])

错误：

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-84-e21592c13279> in <module>
      1 from dask_ml.preprocessing import DummyEncoder
      2 de = DummyEncoder()
----> 3 train_final_cat = de.fit_transform(train_final[categorical_var])

~/env/lib/python3.5/site-packages/sklearn/base.py in fit_transform(self, X, y, **fit_params)
    460         if y is None:
    461             # fit method of arity 1 (unsupervised transformation)
--> 462             return self.fit(X, **fit_params).transform(X)
    463         else:
    464             # fit method of arity 2 (supervised transformation)

~/env/lib/python3.5/site-packages/dask_ml/preprocessing/data.py in fit(self, X, y)
    602 
    603         self.transformed_columns_ = pd.get_dummies(
--> 604             sample, drop_first=self.drop_first
    605         ).columns
    606         return self

~/env/lib/python3.5/site-packages/pandas/core/reshape/reshape.py in get_dummies(data, prefix, prefix_sep, dummy_na, columns, sparse, drop_first, dtype)
    890             dummy = _get_dummies_1d(col[1], prefix=pre, prefix_sep=sep,
    891                                     dummy_na=dummy_na, sparse=sparse,
--> 892                                     drop_first=drop_first, dtype=dtype)
    893             with_dummies.append(dummy)
    894         result = concat(with_dummies, axis=1)

~/env/lib/python3.5/site-packages/pandas/core/reshape/reshape.py in _get_dummies_1d(data, prefix, prefix_sep, dummy_na, sparse, drop_first, dtype)
    978 
    979     else:
--> 980         dummy_mat = np.eye(number_of_cols, dtype=dtype).take(codes, axis=0)
    981 
    982         if not dummy_na:

~/env/lib/python3.5/site-packages/numpy/lib/twodim_base.py in eye(N, M, k, dtype, order)
    184     if M is None:
    185         M = N
--> 186     m = zeros((N, M), dtype=dtype, order=order)
    187     if k >= M:
    188         return m

MemoryError:

任何人都可以在这方面给我一些指导

谢谢

迈克尔

Answer 1

编码虚拟变量是一项非常占用内存的任务，因为您正在为categorical_column的每个唯一值创建一个新列。如果categorical_column是高基数，那么即使单个块也可能爆炸。同样，创建假人也不是“令人尴尬的平行”。因此工作人员不能只是独立地处理每个块。在计算过程中，工作人员需要交流和复制一些数据。

使用虚拟编码器时，dask中的内存错误

1 个答案: