将数据与panda数据框合并时出现内存错误

时间:2018-06-25 10:03:25

标签: python pandas

当我想合并两个数据并且不知道为什么时,我目前正遇到panda的MemoryError。

import pandas
import csv
import numpy as np

ListNames = ["dongsi","tiantan","guanyuan", "wanshouxigong","aotizhongxin","nongzhanguan","wanliu","beibuxinqu","zhiwuyuan","fengtaihuayuan","yungang","gucheng","fangshan","daxing","yizhuang","tongzhou","shunyi","pingchang","mentougou","pinggu","huairou","miyun","yanqin","dingling","badaling","miyunshuiku","donggaocun","yongledian","yufa","liulihe","qianmen","yongdingmennei","xizhimenbei","nansanhuan","dongsihuan"]

order_of_columns_extra = ['stationId','NO2','CO','SO2']

order_of_columns_aq_gp = ['stationId', 'utc_time', 'PM2.5', 'PM10', 'O3', 'temperature', 'pressure', 'humidity', 'wind_direction','wind_speed/kph']

order_of_columns_final = ['stationId', 'utc_time', 'PM2.5', 'PM10', 'O3', 'temperature', 'pressure', 'humidity', 'wind_direction','wind_speed/kph','NO2','CO','SO2']

for i in range(len(ListNames)):
    df_extra = pandas.read_csv(ListNames[i]+'_aq.csv', encoding='utf-8')
    df_aq_gp = pandas.read_csv('normal/'+ListNames[i]+'.csv', encoding = 'utf-8')

    df_Merged = pandas.merge(left = df_extra, right = df_aq_gp, how="left", left_on="stationId", right_on="stationId")

    df_Merged = df_Merged[order_of_columns_final]

    df_Merged.to_csv("WithExtra/"+str(ListNames[i])+".csv", index=False)

我的计算机上有足够的内存,所以可能不是问题 这是给出的完全错误

Traceback (most recent call last):
  File "addExtra.py", line 18, in <module>
    df_Merged = pandas.merge(left = df_extra, right = df_aq_gp, how="left", left_on="stationId", right_on="stationId")
  File "/usr/local/lib/python3.5/dist-packages/pandas/core/reshape/merge.py", line 58, in merge
    return op.get_result()
  File "/usr/local/lib/python3.5/dist-packages/pandas/core/reshape/merge.py", line 596, in get_result
    concat_axis=0, copy=self.copy)
  File "/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py", line 5203, in concatenate_block_managers
    concatenate_join_units(join_units, concat_axis, copy=copy),
  File "/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py", line 5332, in concatenate_join_units
    for ju in join_units]
  File "/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py", line 5332, in <listcomp>
    for ju in join_units]
  File "/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py", line 5632, in get_reindexed_values
    fill_value=fill_value)
  File "/usr/local/lib/python3.5/dist-packages/pandas/core/algorithms.py", line 1379, in take_nd
    out = np.empty(out_shape, dtype=dtype)
MemoryError

0 个答案:

没有答案