Question

希望这是一个允许的SO问题，但我希望得到一些建议，如何转换下面的代码来处理文件中的行，以生成一个使用生成器和产量的数据帧，因为这个实现使用list和append太慢了。

这是我提出的解决方案，但我真的希望避免使用非常慢的列表并追加操作。我希望有一个很酷的发电机和产量解决方案，但还不够舒适，而且还在使用发电机。

文件中的示例行：

"USNC3255","27","US","NC","LANDS   END","72305006","KNJM","KNCA","KNKT","T72305006","","","NCC031","NCZ095","","545","28594","America/New_York","34.65266","-77.07661","7","RDU","893727","
"USNC3256","27","US","NC","LANDSDOWN","72314058","KEHO","KAKH","KIPJ","T72314058","","","NCC045","NCZ068","sc007","517","28150","America/New_York","35.29374","-81.46537","797","CLT","317845","

当前解决方案：

def parse_file(filename):

    newline = []

    with open(filename, 'rb') as f:
        reader = csv.reader(f, quoting=csv.QUOTE_NONE)
        for row in reader:
            newline.append([s.strip('"') for s in row[:-1]])
    df = pd.DataFrame(newline)
    df = df.applymap(lambda x: nan if len(x) == 0 else x).astype(object)
    return df

df = parse_file(filename)

输出只是一个包含23列和两行的数据帧（如果用于上面的采样行）。

Answer 1

您的文件唯一的问题是每行都以,"结尾。这会使解析器混淆。如果您可以删除尾随的逗号和引号，则可以使用常规解析器。

import pandas as pd
from StringIO import StringIO
with open('example.txt') as myfile:
    data = myfile.read().replace(',"\n', '\n')
pd.read_csv(StringIO(data), header=None)

这就是我得到的：

         0   1   2   3            4         5     6     7     8          9   \
0  USNC3255  27  US  NC  LANDS   END  72305006  KNJM  KNCA  KNKT  T72305006   
1  USNC3256  27  US  NC    LANDSDOWN  72314058  KEHO  KAKH  KIPJ  T72314058   

     ...          13     14   15     16                17        18        19  \
0    ...      NCZ095    NaN  545  28594  America/New_York  34.65266 -77.07661   
1    ...      NCZ068  sc007  517  28150  America/New_York  35.29374 -81.46537   

    20   21      22  
0    7  RDU  893727  
1  797  CLT  317845  

[2 rows x 23 columns]

Python Pandas和生成器以CSV格式处理行

1 个答案: