在Python

Question

我有一个680 MB的大型csv文件，我必须在数据框内读取。

我将文件拆分为块，然后将这些块添加到列表中。

然后我尝试使用pd.concat（）创建一个整合的数据框。

我使用下面的代码来实现这个目标：

temp_list = []
chunksize = 10 ** 5

for chunk in pd.read_csv('./data/properties_2016.csv', chunksize=chunksize, low_memory=False):
  temp_list.append(chunk)

properties_df = temp_list[0]

for df in temp_list[1:]:
   properties_df = pd.concat([properties_df, df], ignore_index=True)

我试图通过运行泊坞窗图像来做到这一点。

我遇到了内存错误：

Traceback (most recent call last):
File "dataIngestion.py", line 53, in <module>
properties_df = pd.concat([properties_df, df], ignore_index=True)
File "/usr/local/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 206, in concat
copy=copy)
File "/usr/local/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 266, in __init__
obj._consolidate(inplace=True)
File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 3156, in _consolidate
self._consolidate_inplace()
File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 3138, in _consolidate_inplace
self._protect_consolidate(f)
File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 3127, in _protect_consolidate
result = f()
File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 3136, in f
self._data = self._data.consolidate()
File "/usr/local/lib/python3.6/site-packages/pandas/core/internals.py", line 3573, in consolidate
bm._consolidate_inplace()
File "/usr/local/lib/python3.6/site-packages/pandas/core/internals.py", line 3578, in _consolidate_inplace
self.blocks = tuple(_consolidate(self.blocks))
File "/usr/local/lib/python3.6/site-packages/pandas/core/internals.py", line 4525, in _consolidate
_can_consolidate=_can_consolidate)
File "/usr/local/lib/python3.6/site-packages/pandas/core/internals.py", line 4548, in _merge_blocks
new_values = new_values[argsort]
MemoryError

请在这里帮忙!!

Answer 1

连接DataFrames不会以这种方式工作。我认为link会有所帮助

这是正确的方法

temp_list = []
chunksize = 10 ** 5

for chunk in pd.read_csv('./data/properties_2016.csv', chunksize=chunksize, low_memory=False):
    temp_list.append(chunk)

frames = []
for df in temp_list:
    frames.append(df)
properties_df = pd.concat(frames, ignore_index=True)

我在一个小文件上尝试了它并且工作了，如果你仍然有同样的错误，请告诉我。