Question

我有一个格式为10GB（不能放入RAM）的文件：

Col1,Col2,Col3,Col4
1,2,3,4
34,256,348,
12,,3,4

所以我们有列和缺少值，我想计算第2列和第3列的方法。使用普通的python，我会做类似的事情：

def means(rng):
    s, e = rng

    with open("data.csv") as fd:
        title = next(fd)
        titles = title.split(',')
        print "Means for", ",".join(titles[s:e])

        ret = [0] * (e-s)
        for c, l in enumerate(fd):
            vals = l.split(",")[s:e]
            for i, v in enumerate(vals):
                try:
                    ret[i] += int(v)
                except ValueError:
                    pass

        return map(lambda s: float(s) / (c + 1), ret)

但是我怀疑有一个更快的方法来做numpy（我还是新手）。

Answer 1

Pandas是你最好的朋友：

from pandas.io.parsers import read_csv
from numpy import sum

# Load 10000 elements at a time, you can play with this number to get better
# performance on your machine
my_data = read_csv("data.csv", chunksize=10000)

total = 0
count = 0

for chunk in my_data:
    # If you want to exclude NAs from the average, remove the next line
    chunk = chunk.fillna(0.0)

    total += chunk.sum(skipna=True)
    count += chunk.count()

avg = total / count

col1_avg = avg["Col1"]
# ... etc. ...

Answer 2

尝试：

import numpy
# read from csv into record array
df = numpy.genfromtxt('test.csv',delimiter=',', usecols=(1,2), skip_header=1, usemask=True)
# calc means on columns
ans = numpy.mean(dat, axis=0)

ans.data将包含列的所有方法的数组。

更新问题的编辑

如果您有一个10G文件，您也可以使用numpy来填充它。请参阅此answer。

这样的事情：

sums = numpy.array((0,0))
counts = numpy.array((0,0))
fH = open('test.csv')
fH.readline() # skip header
while True:
    try:
        df = numpy.genfromtxt(itertools.islice(fH, 1000), delimiter=',', usecols=(1,2), usemask=True)
    except StopIteration:
        break       
    sums = sums + numpy.sum(df, 0)
    counts = counts + numpy.sum(df.mask == False, 0)
fH.close()
means = sums / counts

计算意味着从csv与python的numpy

2 个答案: