Question

我使用itertools.combinations()如下：

import itertools
import numpy as np

L = [1,2,3,4,5]
N = 3

output = np.array([a for a in itertools.combinations(L,N)]).T

这使我得到了我需要的输出：

array([[1, 1, 1, 1, 1, 1, 2, 2, 2, 3],
       [2, 2, 2, 3, 3, 4, 3, 3, 4, 4],
       [3, 4, 5, 4, 5, 5, 4, 5, 5, 5]])

我在多处理环境中反复使用这个表达式，我需要它尽可能快。

来自this post我了解基于itertools的代码不是最快的解决方案，使用numpy可能是一种改进，但我不够好numpy优化技巧，以理解和调整在那里编写的迭代代码或提出我自己的优化。

非常感谢任何帮助。

编辑：

L来自一个pandas数据帧，所以它也可以看作是一个numpy数组：

L = df.L.values

Answer 1

这肯定不比def nd_triu_indices(T,N): o=np.array(np.meshgrid(*(np.arange(len(T)),)*N)) return np.array(T)[o[...,np.all(o[1:]>o[:-1],axis=0)]] %timeit np.array(list(itertools.combinations(T,N))).T The slowest run took 4.40 times longer than the fastest. This could mean that an intermediate result is being cached. 100000 loops, best of 3: 8.6 µs per loop %timeit nd_triu_indices(T,N) The slowest run took 4.64 times longer than the fastest. This could mean that an intermediate result is being cached. 10000 loops, best of 3: 52.4 µs per loop更快，但它矢量化numpy：

combinations

不确定这是否是可矢量化的另一种方式，或者如果这里的一个优化向导可以使这种方法更快。

编辑：用另一种方式提出，但仍然不比%timeit np.array(T)[np.array(np.where(np.fromfunction(lambda *i: np.all(np.array(i)[1:]>np.array(i)[:-1], axis=0),(len(T),)*N,dtype=int)))] The slowest run took 7.78 times longer than the fastest. This could mean that an intermediate result is being cached. 10000 loops, best of 3: 34.3 µs per loop：

快

_ =

Answer 2

这是一个比itertools UPDATE稍微快一点的那个：而且其中一个（import numpy as np import itertools import timeit def nump(n, k, i=0): if k == 1: a = np.arange(i, i+n) return tuple([a[None, j:] for j in range(n)]) template = nump(n-1, k-1, i+1) full = np.r_[np.repeat(np.arange(i, i+n-k+1), [t.shape[1] for t in template])[None, :], np.c_[template]] return tuple([full[:, j:] for j in np.r_[0, np.add.accumulate( [t.shape[1] for t in template[:-1]])]]) def nump2(n, k): a = np.ones((k, n-k+1), dtype=int) a[0] = np.arange(n-k+1) for j in range(1, k): reps = (n-k+j) - a[j-1] a = np.repeat(a, reps, axis=1) ind = np.add.accumulate(reps) a[j, ind[:-1]] = 1-reps[1:] a[j, 0] = j a[j] = np.add.accumulate(a[j]) return a def itto(L, N): return np.array([a for a in itertools.combinations(L,N)]).T k = 6 n = 12 N = np.arange(n) assert np.all(nump2(n,k) == itto(N,k)) print('numpy ', timeit.timeit('f(a,b)', number=100, globals={'f':nump, 'a':n, 'b':k})) print('numpy 2 ', timeit.timeit('f(a,b)', number=100, globals={'f':nump2, 'a':n, 'b':k})) print('itertools', timeit.timeit('f(a,b)', number=100, globals={'f':itto, 'a':N, 'b':k}))）实际上要快得多：

k = 3, n = 50
numpy     0.06967267207801342
numpy 2   0.035096961073577404
itertools 0.7981023890897632

k = 3, n = 10
numpy     0.015058324905112386
numpy 2   0.0017436158377677202
itertools 0.004743851954117417

k = 6, n = 12
numpy     0.03546895203180611
numpy 2   0.00997065706178546
itertools 0.05292179994285107

时序：

LoginProcessingFilter

更快的numpy-solution而不是itertools.combinations？

2 个答案: