Numpy __getitem__延迟评估和[-1:]与[slice(-1,None,none)]不一样

时间:2013-01-27 23:22:49

标签: python numpy magic-methods

所以这就是我所假设的两个问题,就像我一样,基本的潜在混淆。我希望没问题。

这里有一些代码:

import numpy as np

class new_array(np.ndarray):

    def __new__(cls, array, foo):
        obj = array.view(cls)
        obj.foo = foo
        return obj

    def __array_finalize__(self, obj):
        print "__array_finalize"
        if obj is None: return
        self.foo = getattr(obj, 'foo', None)

    def __getitem__(self, key):
        print "__getitem__"
        print "key is %s"%repr(key)
        print "self.foo is %d, self.view(np.ndarray) is %s"%(
            self.foo,
            repr(self.view(np.ndarray))
            )
        self.foo += 1
        return super(new_array, self).__getitem__(key)

print "Block 1"
print "Object construction calls"
base_array = np.arange(20).reshape(4,5)
print "base_array is %s"%repr(base_array)
p = new_array(base_array, 0)
print "\n\n"

print "Block 2"
print "Call sequence for p[-1:] is:"
p[-1:]
print "p[-1].foo is %d\n\n"%p.foo

print "Block 3"
print "Call sequence for s = p[-1:] is:"
s = p[-1:]
print "p[-1].foo is now %d"%p.foo
print "s.foo is now %d"%s.foo
print "s.foo + p.foo = %d\n\n"%(s.foo + p.foo)

print "Block 4"
print "Doing q = s + s"
q = s + s
print "q.foo = %d\n\n"%q.foo

print "Block 5"
print "Printing s"
print repr(s)
print "p.foo is now %d"%p.foo
print "s.foo is now %d\n\n"%s.foo

print "Block 6"
print "Printing q"
print repr(q)
print "p.foo is now %d"%p.foo
print "s.foo is now %d"%s.foo
print "q.foo is now %d\n\n"%q.foo

print "Block 7"
print "Call sequence for p[-1]"
a = p[-1]
print "p[-1].foo is %d\n\n"%a.foo

print "Block 8"
print "Call sequence for p[slice(-1, None, None)] is:"
a = p[slice(-1, None, None)]
print "p[slice(None, -1, None)].foo is %d"%a.foo
print "p.foo is %d"%p.foo
print "s.foo + p.foo = %d\n\n"%(s.foo + p.foo)

此代码的输出是

Block 1
Object construction calls
base_array is array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
__array_finalize



Block 2
Call sequence for p[-1:] is:
__array_finalize
p[-1].foo is 0


Block 3
Call sequence for s = p[-1:] is:
__array_finalize
p[-1].foo is now 0
s.foo is now 0
s.foo + p.foo = 0


Block 4
Doing q = s + s
__array_finalize
q.foo = 0


Block 5
Printing s
__getitem__
key is -1
self.foo is 0, self.view(np.ndarray) is array([[15, 16, 17, 18, 19]])
__array_finalize
__getitem__
key is -5
self.foo is 1, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -4
self.foo is 2, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -3
self.foo is 3, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -2
self.foo is 4, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -1
self.foo is 5, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
new_array([[15, 16, 17, 18, 19]])
p.foo is now 0
s.foo is now 1


Block 6
Printing q
__getitem__
key is -1
self.foo is 0, self.view(np.ndarray) is array([[30, 32, 34, 36, 38]])
__array_finalize
__getitem__
key is -5
self.foo is 1, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -4
self.foo is 2, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -3
self.foo is 3, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -2
self.foo is 4, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -1
self.foo is 5, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
new_array([[30, 32, 34, 36, 38]])
p.foo is now 0
s.foo is now 1
q.foo is now 1


Block 7
Call sequence for p[-1]
__getitem__
key is -1
self.foo is 0, self.view(np.ndarray) is array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
__array_finalize
p[-1].foo is 1


Block 8
Call sequence for p[slice(-1, None, None)] is:
__getitem__
key is slice(-1, None, None)
self.foo is 1, self.view(np.ndarray) is array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
__array_finalize
p[slice(None, -1, None)].foo is 2
p.foo is 2
s.foo + p.foo = 3

请注意两件事:

  1. p[-1:]的调用不会导致对new_array.__getitem__的调用。如果将p[-1:]替换为p[0:]p[0:-1]等内容,则属实......但p[-1]p[slice(-1, None, None)]之类的语句会导致调用new_array.__getitem__。对于p[-1:] + p[-1:]s = p[-1]等语句也是如此,但对于print s这样的语句则不然。您可以通过查看上面给出的“块”来看到这一点。

  2. 变量foo在调用new_array.__getitem__期间已正确更新(参见第5和第6块),但在new_array.__getitem__的评估完成后不正确(请再次参阅,第5和第6栏)。我还应该补充说,用return super(new_array, self).__getitem__(key)替换行return new_array(np.array(self.view(np.ndarray)[key]), self.foo)也不起作用。以下块是输出中唯一的差异。

    Block 5
    Printing s
    __getitem__
    key is -1
    self.foo is 0, self.view(np.ndarray) is array([[15, 16, 17, 18, 19]])
    __array_finalize__
    __getitem__
    key is -5
    self.foo is 1, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -4
    self.foo is 2, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -3
    self.foo is 3, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -2
    self.foo is 4, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -1
    self.foo is 5, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    new_array([[15, 16, 17, 18, 19]])
    p.foo is now 0
    s.foo is now 1
    
    
    Block 6
    Printing q
    __getitem__
    key is -1
    self.foo is 0, self.view(np.ndarray) is array([[30, 32, 34, 36, 38]])
    __array_finalize__
    __getitem__
    key is -5
    self.foo is 1, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -4
    self.foo is 2, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -3
    self.foo is 3, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -2
    self.foo is 4, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    __getitem__
    key is -1
    self.foo is 5, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
    __array_finalize__
    __array_finalize__
    __array_finalize__
    new_array([[30, 32, 34, 36, 38]])
    p.foo is now 0
    s.foo is now 1
    q.foo is now 1
    

    现在包含对new_array.__array_finalize__的过多调用,但没有 使用变量foo更改“问题”。

  3. 我希望像p[-1:]这样的new_arrayp.foo = 0对象的调用会导致此语句p.foo == 1返回True。显然情况并非如此,即使在foo调用期间__getitem__被正确更新,因为p[-1:]之类的语句会导致对__getitem__的大量调用(一旦考虑延迟评估)。此外,调用p[-1:]p[slice(-1, None, None)]会导致foo的不同值(如果计数的内容正常工作)。在前一种情况下,foo会添加5,而在后一种情况下,foo会添加1

  4. 问题

    虽然对numpy数组切片的延迟评估在评估我的代码时不会引起问题,但使用pdb调试我的一些代码却是一个巨大的痛苦。基本上,语句在运行时和pdb中的评估方式不同。我认为这不好。这就是我偶然发现这种行为的原因。

    我的代码使用__getitem__的输入来评估应返回的对象类型。在某些情况下,它返回一个相同类型的新实例,在其他情况下,它返回一个其他类型的新实例,在其他情况下,它返回一个numpy数组,标量或浮点数(取决于底层numpy数组认为是正确的) )。我使用传递给__getitem__的密钥来确定要返回的正确对象是什么。但是如果用户通过切片,我不能这样做,例如像p[-1:]这样的东西,因为该方法只获得单独的索引,例如好像用户写了p[4]那么,如果我的numpy子类的key中的__getitem__没有反映用户是否正在请求切片,p[-1:]给出,或者只是一个条目,我该如何做,由p[4]提供?

    作为一个侧面点,numpy indexing文档暗示了切片对象,例如slice(start, stop, step)将与start:stop:step等语句一样对待。这让我觉得我错过了一些非常基本的东西。暗示这种情况的句子很早就出现了:

      

    当obj是切片对象(由start:stop:step表示法构造)时,会发生基本切片   括号内),整数或切片对象和整数的元组。

    我不禁觉得同样的基本错误也是我认为self.foo += 1行应该计算用户请求切片的次数或{的实例的元素的原因{1}}(而不是切片中“元素”的数量)。 这两个问题实际上是否相关,如果是这样的话?

2 个答案:

答案 0 :(得分:8)

你确实被一个讨厌的小虫咬了一口。知道我不是唯一一个人,这是一种解脱!幸运的是,它很容易解决。只需在课堂上添加以下内容即可。这实际上是我几个月前写的一些代码的复制粘贴,文档字符串告诉我们发生了什么,但你也可能想要阅读the python docs

def __getslice__(self, start, stop) :
    """This solves a subtle bug, where __getitem__ is not called, and all
    the dimensional checking not done, when a slice of only the first
    dimension is taken, e.g. a[1:3]. From the Python docs:
       Deprecated since version 2.0: Support slice objects as parameters
       to the __getitem__() method. (However, built-in types in CPython
       currently still implement __getslice__(). Therefore, you have to
       override it in derived classes when implementing slicing.)
    """
    return self.__getitem__(slice(start, stop))

答案 1 :(得分:0)

使用isinstance方法检查切片类型。

from __future__ import print_function

class SliceExample(object):
    def __getitem__(self, key):
        if isinstance(key, slice):
            return key.start, key.stop
        return key

sl = SliceExample()

print(repr(sl[1]))
print(repr(sl[1:2]))