Question

有没有办法将标题行添加到CSV而不将CSV加载到python的内存中？我有一个18GB的CSV，我想添加一个标题，我所见的所有方法都需要将CSV加载到内存中，这显然是不可行的。

Answer 1

您需要重写整个文件。最简单的是不使用python

echo 'col1, col2, col2,... ' > out.csv
cat in.csv >> out.csv

基于Python的解决方案可以在更高的级别上运行，并且速度会慢很多。毕竟18GB是很多数据。最好使用操作系统功能，这将是最快的。

Answer 2

只使用csv模块对行进行迭代的事实，因此它永远不会将整个文件加载到内存中

import csv

with open("huge_csv.csv") as fr, open("huge_output.csv","w",newline='') as fw:
    cr = csv.reader(fr)
    cw = csv.writer(fw)
    cw.writerow(["title1","title2","title3"])
    cw.writerows(cr)

使用writerows确保非常好的速度。记忆在这里幸免。一切都是逐行完成的。由于数据已正确处理，您甚至可以更改输出文件中的分隔符和/或引号。

Answer 3

以下是对于具有10 ^ 6行和10列（n = 50）的~200 MB CSV文件的三种建议解决方案的比较。对于较大和较小的文件（10 MB到8 GB），该比率保持大致相同。

cp：shutil：csv_reader 1:10:55

即。使用内置cp函数比使用Python的csv模块快大约55倍。

电脑：

普通硬盘
Python 3.5.2 64位
Ubuntu 16.04
i7-3770

import csv
import random
import shutil
import time
import subprocess

rows = 1 * 10**3
cols = 10
repeats = 50

shell_script = '/tmp/csv.sh'
input_csv = '/tmp/temp.csv'
output_csv = '/tmp/huge_output.csv'
col_titles = ['titles_' + str(i) for i in range(cols)]

with open(shell_script, 'w') as f:
    f.write("#!/bin/bash\necho '{0}' > {1}\ncat {2} >> {1}".format(','.join(col_titles), output_csv, input_csv))

with open(shell_script, 'w') as f:
    f.write("echo '{0}' > {1}\ncat {2} >> {1}".format(','.join(col_titles), output_csv, input_csv))
subprocess.call(['chmod', '+x', shell_script])

run_times = dict([
    ('csv_writer', list()),
    ('external', list()),
    ('shutil', list())
])

def random_csv():
    with open(input_csv, 'w') as csvfile:
        csv_writer = csv.writer(csvfile, delimiter=',')
        for i in range(rows):
            csv_writer.writerow([str(random.random()) for i in range(cols)])
    with open(output_csv, 'w'):
        pass

for r in range(repeats):
    random_csv()
    #http://stackoverflow.com/a/41982368/2776376
    start_time = time.time()
    with open(input_csv) as fr, open(output_csv, "w", newline='') as fw:
        cr = csv.reader(fr)
        cw = csv.writer(fw)
        cw.writerow(col_titles)
        cw.writerows(cr)
    run_times['csv_writer'].append(time.time() - start_time)

    random_csv()
    #http://stackoverflow.com/a/41982383/2776376
    start_time = time.time()
    subprocess.call(['bash', shell_script])
    run_times['external'].append(time.time() - start_time)

    random_csv()
    #http://stackoverflow.com/a/41982383/2776376
    start_time = time.time()
    with open('header.txt', 'w') as header_file:
        header_file.write(','.join(col_titles))

    with open(output_csv, 'w') as new_file:
        with open('header.txt', 'r') as header_file, open(input_csv, 'r') as main_file:
            shutil.copyfileobj(header_file, new_file)
            shutil.copyfileobj(main_file, new_file)
    run_times['shutil'].append(time.time() - start_time)

    print('#'*20)
    for key in run_times:
        print('{0}: {1:.2f} seconds'.format(key, run_times[key][-1]))

print('#'*20)
print('Averages')
for key in run_times:
    print('{0}: {1:.2f} seconds'.format(key, sum(run_times[key])/len(run_times[key])))

如果您真的想在Python中执行此操作，可以先创建头文件，然后通过shutil.copyfileobj将其与第二个文件合并。

import shutil
with open('header.txt', 'w') as header_file:
    header_file.write('col1;col2;col3')

with open('new_file.csv', 'w') as new_file:
    with open('header.txt', 'r') as header_file, open('main.csv', 'r') as main_file:
        shutil.copyfileobj(header_file, new_file)
        shutil.copyfileobj(main_file, new_file)

将标头添加到CSV而不加载CSV

3 个答案: