Question

我是Python的新手，我正在寻找使用锯齿状数组进行操作的最有效方法。

我有一个像这样的锯齿状数组：

A = array([[array([1, 2, 3]), array([4, 5])],[array([6, 7, 8, 9]), array([10])]], dtype=object)

我希望能够做到这样的事情：

A=A[A>4]
B=A+A

显然python对于使用numpy数组进行此类操作非常有效，但不幸的是我需要为锯齿状数组执行此操作，而我在Python中找不到这样的对象。它是否存在于Python中，或者是否存在允许使用锯齿状数组进行高效操作的库？

对于我给出的例子，这里是我喜欢的输出：

A = array([[array([]), array([5])],[array([6, 7, 8, 9]), array([10])]], dtype=object)
B = array([[array([]), array([10])],[array([12, 14, 16, 18]), array([20])]], dtype=object)

但也许Python工作的方式根本无法像锯齿阵列那样使用锯齿状数据进行高效操作，我不知道细节。

Answer 1

你的数组是2x2：

In [298]: A
Out[298]: 
array([[array([1, 2, 3]), array([4, 5])],
       [array([6, 7, 8, 9]), array([10])]], dtype=object)

虽然A+A有效，但尚未对此类数组实现布尔测试：

In [299]: A>4
...
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

我要压扁A因为它更容易与列表操作进行比较：

In [301]: A1=A.flatten()

In [303]: A1+A1
Out[303]: 
array([array([2, 4, 6]), array([ 8, 10]), array([12, 14, 16, 18]),
       array([20])], dtype=object)

In [304]: [a+a for a in A1]
Out[304]: [array([2, 4, 6]), array([ 8, 10]), array([12, 14, 16, 18]), array([20])]

In [305]: timeit A1+A1
100000 loops, best of 3: 6.85 µs per loop

In [306]: timeit [a+a for a in A1]
100000 loops, best of 3: 9.09 µs per loop

数组操作比列表理解快一点。但是，如果我首先将数组转换为列表：

In [307]: A1l=A1.tolist()

In [308]: A1l
Out[308]: [array([1, 2, 3]), array([4, 5]), array([6, 7, 8, 9]), array([10])]

In [309]: timeit [a+a for a in A1l]
100000 loops, best of 3: 5.2 µs per loop

时间有所改善。这很好地表明A1+A1（或甚至A+A）正在使用类似的迭代。

因此，执行A,B计算的直接方法是

In [310]: A2=[a[a>4] for a in A1]
In [311]: B=[a+a for a in A2]
In [312]: B
Out[312]: [array([], dtype=int32), array([10]), array([12, 14, 16, 18]), array([20])]

（我们可以根据需要转换为数组和列表）。

numpy数组将其数据存储为平面数据缓冲区，并使用shape和strides属性快速计算任何元素的位置，而不管维度如何。快速数组操作使用编译代码，快速遍历参数的数据库，逐个元素（或其他组合）执行操作。

dtype object数组也有扁平数据缓冲区，但元素是指向其他地方的列表或数组的指针。因此，虽然它可以快速索引单个元素，但它仍然必须执行Python调用来访问数组。因此，特别是当数组为1d时，它实际上与具有相同指针的平面列表相同。

多维对象数组比嵌套列表更好。您可以重塑它们，访问元素（A[1,3] v Al[1][3]），转置它们等等。但是当涉及遍历所有子阵列时，它们并没有带来太大的好处。

再看一下你的2D阵列：

In [315]: timeit A+A
100000 loops, best of 3: 6.93 µs per loop  # 6.85 for A1+A1 (above)

In [316]: timeit [[j+j for j in i] for i in A]
100000 loops, best of 3: 17.1 µs per loop

In [317]: Al = A.tolist()

In [318]: timeit [[j+j for j in i] for i in Al]
100000 loops, best of 3: 7.01 µs per loop    # 5.2 for A1l flat list

基本上同时对数组进行求和并迭代等效的嵌套列表。

Answer 2

The performance of numpy jagged array may not be optimal, but there are enough reasons to believe that it should be much better than using python nested list. As explained in your earlier post:

On principle you should have some performance bonus because every element is a numpy array. So you just need a 2 dimensional loop rather than a 3D loop (if you store every number in nested lists). Also it always saves you lots of memory allocation time to avoid using python list.

Here is a simple test:

import time,sys,random
import numpy as np
rand = np.random.rand
L = np.array([[rand(100), rand(200)],[rand(400), rand(300)]], dtype=object)
L1 = [random.random() for i in range(1000)]
arrFunc = np.vectorize(lambda x:x[x>0.3],otypes=[np.ndarray])

start = time.time()
if sys.argv[1]=='np':
  for i in range(100000):
    B=i*L
else:
  for i in range(100000):
    B=[i*x for x in L1]

end = time.time()
print ('Arithmetic Op: ', end-start)


start = time.time()
if sys.argv[1]=='np':
  for i in range(100000):
    B=arrFunc(L)
else:
  for i in range(100000):
    B=[x for x in L1 if x<0.3]
end = time.time()
print ('Indexing       ', end-start)

Result:

> python testNpJarray.py np
Arithmetic Op:  3.9719998836517334
Indexing        8.079999923706055

> python testNpJarray.py list
Arithmetic Op:  53.289000034332275
Indexing        52.10899996757507

This test may not be quite fare because the outter numpy array is quite small, you are welcome to change the size to fit into your application and tell us the results.

python jagged数组操作效率

2 个答案: