Question

有没有办法将pandas数据帧拆分成多个受内存使用限制的数据帧？

Answer 1

def split_dataframe(df, size):

    # size of each row
    row_size = df.memory_usage().sum() / len(df)

    # maximum number of rows of each segment
    row_limit = size // row_size

    # number of segments
    seg_num = (len(df) + row_limit - 1) // row_limit

    # split df
    segments = [df.iloc[i*row_limit : (i+1)*row_limit] for i in range(seg_num)]

    return segments

Answer 2

最简单的方法是，数据帧的列是一致的数据类型（即不是对象）。这是一个如何解决这个问题的例子。

import numpy as np
import pandas as pd
from __future__ import division

df = pd.DataFrame({'a': [1]*100, 'b': [1.1, 2] * 50, 'c': range(100)})

# calculate the number of bytes a row occupies
row_bytes = df.dtypes.apply(lambda x: x.itemsize).sum()

mem_limit = 1024

# get the maximum number of rows in a segment
max_rows = mem_limit / row_bytes

# get the number of dataframes after splitting
n_dfs = np.ceil(df.shape[0] / max_rows)

# get the indices of the dataframe segments
df_segments = np.array_split(df.index, n_dfs)

# create a list of dataframes that are below mem_limit
split_dfs = [df.loc[seg, :] for seg in df_segments]

split_dfs

此外，如果您可以按列而不是行进行拆分，则pandas可以使用方便的memory_usage方法。

Pandas数据帧按内存使用情况分割

2 个答案: