Question

我将文件合并到4个文件夹中。在这4个文件夹中，我将80 .dbf个文件合并在一起，每个文件都是35兆字节。我使用以下代码：

import os
import pandas as pd
from simpledbf import Dbf5

list1=[]   
folders=r'F:\dbf_tables'
out=r'F:\merged'
if not os.path.isdir(out):
    os.mkdir(out)
for folder in os.listdir(folders):
    if not os.path.isdir(os.path.join(out,folder)):
        os.mkdir(os.path.join(out,folder))
    for f in os.listdir(os.path.join(folders,folder)):
         if '.xml' not in f:
            if '.cpg' not in f:
                 table=Dbf5(os.path.join(folders,folder,f))
                 df=table.to_dataframe()
                 list1.append(df)
                 dfs = reduce(lambda left,right: pd.merge(left,right,on=['POINTID'],how='outer',),list1)
                 dfs.to_csv(os.path.join(out,folder,'combined.csv'), index=False)

在运行代码后几乎立即收到此错误：

Traceback (most recent call last):

  File "<ipython-input-1-77eb6fd0cda7>", line 1, in <module>
    runfile('F:/python codes/prelim_codes/raster_to_point.py', wdir='F:/python codes/prelim_codes')

  File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "F:/python codes/prelim_codes/raster_to_point.py", line 66, in <module>
    dfs = reduce(lambda left,right: pd.merge(left,right,on=['POINTID'],how='outer',),list1)

  File "F:/python codes/prelim_codes/raster_to_point.py", line 66, in <lambda>
    dfs = reduce(lambda left,right: pd.merge(left,right,on=['POINTID'],how='outer',),list1)

  File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 39, in merge
    return op.get_result()

  File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 217, in get_result
    join_index, left_indexer, right_indexer = self._get_join_info()

  File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 353, in _get_join_info
    sort=self.sort, how=self.how)

  File "C:\Users\spotter\AppData\Local\Continuum\Anaconda_64\lib\site-packages\pandas\tools\merge.py", line 559, in _get_join_indexers
    return join_func(lkey, rkey, count, **kwargs)

  File "pandas\src\join.pyx", line 160, in pandas.algos.full_outer_join (pandas\algos.c:61256)

MemoryError

但只有30％的记忆被使用，这几乎就是基线。

编辑：

我只挑出了2个文件并尝试使用以下方法进行合并：

merge=pd.merge(df1,df2, on=['POINTID'], how='outer')

并且仍然会出现内存错误，这种情况很奇怪。

当我在32位Anaconda中运行相同的东西时，我得到ValueError: negative dimensions are not allowed

编辑：

整个问题源于此处的解决方案： Value Error: negative dimensions are not allowed when merging

Answer 1

根据评论

已编辑：

试试这个（仅使用一个带有逻辑if条件的and语句就足够了）：

import os
import pandas as pd
from simpledbf import Dbf5

folders = r'F:\dbf_tables'
out = r'F:\merged'

if not os.path.isdir(out):
    os.mkdir(out)

for folder in os.listdir(folders):
    if not os.path.isdir(os.path.join(out, folder)):
        os.mkdir(os.path.join(out, folder))

    # Initialize empty dataframe by folders
    dfs = pd.DataFrame(columns=['POINTID'])

    for f in os.listdir(os.path.join(folders, folder)):
         if ('.xml' not in f) and ('.cpg' not in f):
             table = Dbf5(os.path.join(folders, folder, f))
             df = table.to_dataframe()

             # Merge actual dataframe to result dataframe
             dfs = dfs.merge(df, on=['POINTID'], how='outer')

    # Save results by folder
    dfs.to_csv(os.path.join(out, folder, 'combined.csv'), index=False)

内存错误，即使RAM是免费的

1 个答案: