Question

我有一个相对较大（约3GB，3百万条目）的子列表列表，其中每个子列表包含一组标签。这是一个非常简单的例子：

tag_corpus = [['cat', 'fish'], ['cat'], ['fish', 'dog', 'cat']]  

unique_tags = ['dog', 'cat', 'fish'] 
co_occurences = {key:Counter() for key in unique_tags}

for tags in tag_corpus: 
    tallies = Counter(tags)
    for key in tags: 
        co_occurences[key] = co_occurences[key] + tallies

这就像魅力一样，但它的实际数据集的速度很慢，因为它具有非常大的子列表（大约30K的唯一标记）。任何python专业人员都知道如何加速这件事吗？

Answer 1

此可能更快。你必须要衡量。

from collections import Counter
from collections import defaultdict

tag_corpus = [['cat', 'fish'], ['cat'], ['fish', 'dog', 'cat']]

co_occurences = defaultdict(Counter)
for tags in tag_corpus:
    for key in tags:
        co_occurences[key].update(tags)
unique_tags = sorted(co_occurences)

print co_occurences
print unique_tags

Answer 2

我只是在喋喋不休，不期望最终得到更高效的东西，但是拥有100000只猫，狗和鱼，这个速度要快得多，时间为0.07秒而不是1.25秒。

我试图找到一个更短的解决方案，但事实证明这种方式是最快的，即使它看起来很简单：）

occurances = {}
for tags in tag_corpus:
    for key in tags:
        for key2 in tags:
            try:
                occurances[key][key2] += 1
            except KeyError:
                try:
                    occurances[key][key2] = 1
                except KeyError:
                    occurances[key] = {key2: 1}

Answer 3

您可以尝试使用 defaultdict 进行梳理以避免在开始时使用Peters回答的逻辑进行初始化，运行时间会明显加快：

In [35]: %%timeit
co_occurences = defaultdict(Counter)
for tags in tag_corpus:
    for key in tags:
        co_occurences[key].update(tags)
   ....: 

1 loop, best of 3: 513 ms per loop

In [36]: %%timeit
occurances = {k1: defaultdict(int) for k1 in unique_tags}
for tags in tag_corpus:
    for key in tags:
        for key2 in tags:
            occurances[key][key2] += 1
   ....: 
10 loops, best of 3: 65.7 ms per loop

In [37]: %%timeit
   ....: co_occurences = {key:Counter() for key in unique_tags}
   ....: for tags in tag_corpus: 
   ....:     tallies = Counter(tags)
   ....:     for key in tags: 
   ....:         co_occurences[key] = co_occurences[key] + tallies
   ....: 
 1 loop, best of 3: 1.13 s per loop
    In [38]: %%timeit
   ....: occurances = defaultdict(lambda: defaultdict(int))
   ....: for tags in tag_corpus:
   ....:     for key in tags:
   ....:         for key2 in tags:
   ....:             occurances[key][key2] += 1
   ....: 
10 loops, best of 3: 66.5 ms per loop

至少在python2中， Counter dict实际上并不是获取计数的最快方法，但是 defaultdict 即使使用lambda也是如此。

即使滚动自己的计数功能也会更快：

def count(x):
    d = defaultdict(int)
    for ele in x:
        d[ele] += 1
    return d

没有最快但仍然非常好的快：

In [42]: %%timeit
   ....: co_occurences = {key: defaultdict(int) for key in unique_tags}
   ....: for tags in tag_corpus:
   ....:     tallies = count(tags)
   ....:     for key in tags:
   ....:         for k, v in tallies.items():
   ....:             co_occurences[key][k] += v
   ....: 

10 loops, best of 3: 164 ms per loop

如果你想要更多的加速，那么一些cython可能会有很长的路要走。

如何有效地计算python列表中的共现

3 个答案: