基于答案here，似乎没有一种简单的方法可以用生成器中的数据填充2D numpy数组。
但是，如果有人能想出一种矢量化或以其他方式加速以下功能的方法，我将不胜感激。
这里的区别在于我想要批量处理生成器中的值，而不是在内存中创建整个数组。我能想到的唯一方法是使用for循环。
import numpy as np
from itertools import permutations
permutations_of_values = permutations(range(1,20), 7)
def array_from_generator(generator, arr):
"""Fills the numpy array provided with values from
the generator provided. Number of columns in arr
must match the number of values yielded by the
generator."""
count = 0
for row in arr:
try:
item = next(generator)
except StopIteration:
break
row[:] = item
count += 1
return arr[:count,:]
batch_size = 100000
empty_array = np.empty((batch_size, 7), dtype=int)
batch_of_values = array_from_generator(permutations_of_values, empty_array)
print(batch_of_values[0:5])
输出：
[[ 1 2 3 4 5 6 7]
[ 1 2 3 4 5 6 8]
[ 1 2 3 4 5 6 9]
[ 1 2 3 4 5 6 10]
[ 1 2 3 4 5 6 11]]
速度测试：
%timeit array_from_generator(permutations_of_values, empty_array)
10 loops, best of 3: 137 ms per loop
此外：
正如@COLDSPEED（感谢）所建议的，这是一个使用列表从发生器收集数据的版本。它的速度大约是上面代码的两倍。任何人都可以改进：
permutations_of_values = permutations(range(1,20), 7)
def array_from_generator2(generator, rows=batch_size):
"""Creates a numpy array from a specified number
of values from the generator provided."""
data = []
for row in range(rows):
try:
data.append(next(generator))
except StopIteration:
break
return np.array(data)
batch_size = 100000
batch_of_values = array_from_generator2(permutations_of_values, rows=100000)
print(batch_of_values[0:5])
输出：
[[ 1 2 3 4 5 6 7]
[ 1 2 3 4 5 6 8]
[ 1 2 3 4 5 6 9]
[ 1 2 3 4 5 6 10]
[ 1 2 3 4 5 6 11]]
速度测试：
%timeit array_from_generator2(permutations_of_values, rows=100000)
10 loops, best of 3: 85.6 ms per loop
答案 0 :(得分：3)
您可以在基本恒定的时间内计算出前方的尺寸。就这样做，并使用numpy.fromiter：
In [1]: import math, from itertools import permutations, chain
In [2]: def n_chose_k(n, k, fac=math.factorial):
...: return fac(n)/fac(n-k)
...:
In [3]: def permutations_to_array(r, k):
...: n = len(r)
...: size = int(n_chose_k(n, k))
...: it = permutations(r, k)
...: arr = np.fromiter(chain.from_iterable(it),
...: count=size, dtype=int)
...: arr.size = size//k, k
...: return arr
...:
In [4]: arr = permutations_to_array(range(1,20), 7)
In [5]: arr.shape
Out[5]: (36279360, 7)
In [6]: arr[0:5]
Out[6]:
array([[ 1, 2, 3, 4, 5, 6, 7],
[ 1, 2, 3, 4, 5, 6, 8],
[ 1, 2, 3, 4, 5, 6, 9],
[ 1, 2, 3, 4, 5, 6, 10],
[ 1, 2, 3, 4, 5, 6, 11]])
只要r仅限于len的序列，此功能就会有效。
编辑添加我为batchsize*k块生成器制作的实现，并带有修剪选项！
import math
from itertools import repeat, chain
import numpy as np
def n_chose_k(n, k, fac=math.factorial):
return fac(n)/fac(n-k)
def permutations_in_batches(r, k, batchsize=None, fill=0, dtype=int, trim=False):
n = len(r)
size = int(n_chose_k(n, k))
if batchsize is None or batchsize > size:
batchsize = size
perms = chain.from_iterable(permutations(r, k))
count = batchsize*k
remaining = size - count
while remaining > 0:
current = np.fromiter(perms, count=count, dtype=dtype)
current.shape = batchsize, k
yield current
remaining -= count
if remaining: # remaining is negative
remaining = -remaining
if not trim:
padding = repeat(fill, remaining)
finalcount = count
finalshape = batchsize, k
else:
q = remaining//k # always divisible q%k==0
finalcount = q*k
padding = repeat(fill, remaining)
finalshape = q, k
current = np.fromiter(chain(perms, padding), count=finalcount, dtype=dtype)
current.shape = finalshape
else: # remaining is 0
current = np.fromiter(perms, count=batchsize, dtype=dtype)
current.shape = batchsize, k
yield current