Question

我试图在某些数据不适合内存时使用memmap，并使用memmap的功能来欺骗代码，认为它只是一个ndarray。为了进一步扩展这种使用memmap的方式，我想知道是否可以重载memmap的解除引用操作符来删除memmap文件。

例如：

from tempfile import mkdtemp
import os.path as path
filename = path.join(mkdtemp(), 'tmpfile.dat')
{
    out = np.memmap(filename, dtype=a.dtype, mode='w+', shape=a.shape)
}
# At this point out is out of scope, so the overloaded 
# dereference function would delete tmpfile.dat

这听起来可行/已经完成了吗？我有没有想到的东西？

谢谢！

Answer 1

在np.memmap打开文件后删除该文件在关闭文件描述符的最后一次引用后，系统将删除该文件。

python临时文件的工作方式与此类似，可以非常方便地与with上下文管理器结构一起使用：

with tempfile.NamedTemporaryFile() as f:
    # file gone now from the filesystem 
    # but f still holds a reference so it still exists and uses space (see /prof<pid>/fd)
    # open it again (will not work on windows)
    x = np.memmap(f.name, dtype=np.float64, mode='w+', shape=(3,4))
# file path is unlinked but exists on disk with the open file reference in x
del x
# now all references are gone and the file is properly deleted

Answer 2

一种情况，如果我们不想使用，而只是有一些类可以为我们处理：

class tempmap(np.memmap):
    """
    Extension of numpy memmap to automatically map to a file stored in temporary directory.
    Usefull as a fast storage option when numpy arrays become large and we just want to do some quick experimental stuff.
    """
    def __new__(subtype, dtype=np.uint8, mode='w+', offset=0,
                shape=None, order='C'):
        ntf = tempfile.NamedTemporaryFile()
        self = np.memmap.__new__(subtype, ntf, dtype, mode, offset, shape, order)
        self.temp_file_obj = ntf
        return self

    def __del__(self):
        if hasattr(self,'temp_file_obj') and self.temp_file_obj is not None:
            self.temp_file_obj.close()
            del self.temp_file_obj

def np_as_tmp_map(nparray):
    tmpmap = tempmap(dtype=nparray.dtype, mode='w+', shape=nparray.shape)
    tmpmap[...] = nparray
    return tmpmap


def test_memmap():
    """Test, that deleting a temp memmap also deletes the file."""
    x = np_as_tmp_map(np.zeros(10, 10), np.float))
    name = copy(x.temp_file_obj.name)
    del x
    x = None
    assert not os.path.isfile(name)

在Python中，当不再引用memmap对象时，是否可以重载Numpy的memmap以删除自身？

2 个答案: