Question

我有一个字典列表，键有'a'，'n'，'o'，'u'。有没有办法加快这个计算，例如使用NumPy？列表中有数万个项目。

数据是从数据库中提取的，所以我最初必须以字典列表的形式存在。

x = n = o = u = 0
for entry in indata:
    x += (entry['a']) * entry['n']  # n - number of data points
    n += entry['n']
    o += entry['o']
    u += entry['u']

    loops += 1

average = int(round(x / n)), n, o, u

Answer 1

我怀疑这会更快，但我认为它是timeit的候选人......

from operator import itemgetter
x = n = o = u = 0
items = itemgetter('a','n','o','u')
for entry in indata:
    A,N,O,U = items(entry)
    x += A*N  # n - number of data points
    n += N
    o += O    #don't know what you're doing with O or U, but I'll leave them
    u += U

average = int(round(x / n)), n, o, u

至少，它保存了entry['n']的查找，因为我现在已将其保存到变量

Answer 2

您可以尝试这样的事情：


mean_a = np.sum(np.array([d['a'] for d in data]) * np.array([d['n'] for d in data])) / len(data)

编辑：实际上，@ mgilson的上述方法更快：


import numpy as np
from operator import itemgetter
from pandas import *

data=[]
for i in range(100000):
    data.append({'a':np.random.random(), 'n':np.random.random(), 'o':np.random.random(), 'u':np.random.random()})

def func1(data):
    x = n = o = u = 0
    items = itemgetter('a','n','o','u')
    for entry in data:
        A,N,O,U = items(entry)
        x += A*N  # n - number of data points
        n += N
        o += O    #don't know what you're doing with O or U, but I'll leave them
        u += U

    average = int(round(x / n)), n, o, u
    return average


def func2(data):
    mean_a = np.sum(np.array([d['a'] for d in data]) * np.array([d['n'] for d in data])/len(data)
    return (mean_a, 
                np.sum([d['n'] for d in data]), 
                np.sum([d['o'] for d in data]), 
                np.sum([d['u'] for d in data])
               )

def func3(data):
    dframe = DataFrame(data)
    return np.sum((dframe["a"]*dframe["n"])) / dframe.shape[0], np.sum(dframe["n"]), np.sum(dframe["o"]), np.sum(dframe["u"])

In [3]: %timeit func1(data)
10 loops, best of 3: 59.6 ms per loop

In [4]: %timeit func2(data)
10 loops, best of 3: 138 ms per loop

In [5]: %timeit func3(data)
10 loops, best of 3: 129 ms per loop

如果您正在对数据进行其他操作，我肯定会考虑使用Pandas包。它的DataFrame对象与您正在使用的字典列表很匹配。我认为大部分开销是将数据导入numpy数组或DataFrame对象的IO操作。

Answer 3

如果您要做的就是获得平均值，为什么不

sum_for_average = math.fsum(your_item)
average_of_list = sum_for_average / len(your_item)

根本没有任何关于numpy的事。

从字典列表中优化平均计算

3 个答案:

编辑：实际上，@ mgilson的上述方法更快：