熊猫:合并两个数据框时抛出MemoryError异常

时间:2019-09-04 07:12:27

标签: python pandas dataframe

我正在两个数据帧之间进行合并,其中一个最大为392M行,而另一个则小得多(<1M行),这会引发如下错误:

  File "foo.py", line 341, in foo
    hist_stat = dim_adj_ctr_item.merge(sku_stat, on =['item_id'], how = 'left', suffixes = ['', '_stat'])
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/frame.py", line 6868, in merge
    copy=copy, indicator=indicator, validate=validate)
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py", line 48, in merge
    return op.get_result()
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py", line 560, in get_result
    concat_axis=0, copy=self.copy)
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 2061, in concatenate_block_managers
    concatenate_join_units(join_units, concat_axis, copy=copy),
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/internals/concat.py", line 242, in concatenate_join_units
    for ju in join_units]
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/internals/concat.py", line 242, in <listcomp>
    for ju in join_units]
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/internals/concat.py", line 225, in get_reindexed_values
    fill_value=fill_value)
  File "/.../anaconda3/lib/python3.7/site-packages/pandas/core/algorithms.py", line 1651, in take_nd
    out = np.empty(out_shape, dtype=dtype)
MemoryError

有什么方法可以使该语句生效?

0 个答案:

没有答案