Question

我正在尝试编写一个python类，使用multiprocessing.Pool和threading.Lock并行读取大量图像。我的方法是创建一个线程池，每个线程将读取一个图像并附加到具有列表类型的成员变量。它还将提供在完成所有图像读取后获取列表的功能。

class ReadFilePool(object):
    # filenames contains a list of image absolute paths
    def __init__(self, filenames):
        self.filenames = filenames
        self.images = []
        self.lock = threading.Lock()
        self.pool = Pool(processes=len(self.filenames))
        self.pool.map(self.read_file, [filename for filename in self.filenames])

    def read_file(self, filename):
        image = get_image(filename)
        self.lock.acquire()
        self.images.append(image)
        self.lock.release()

    def get_images(self):
        images = None
        self.lock.acquire()
        if len(self.filenames) == len(self.images):
            images = self.images
        self.lock.release()
        return images

然后我将尝试循环并检查get_images是否不是None并处理图像，例如

images = []
completed = False
pool = ReadFilePool(filenames)
while not completed:
    images = pool.get_images()
    completed = (None == images)
# ...process the images

我尝试使用以下方法，但仍然遇到像TypeError: can't pickle _thread.lock objects

这样的pickle错误

方法1：__setstate__ and __getstate__

方法2：__call__

不幸的是，我对python multithreading和Lock并不太熟悉，并且遇到了很少与pickle相关的错误。请建议使用这些课程的正确方法。

如何在类中使用multiprocessing.Pool和threading.Lock

0 个答案: