多处理Python

Question

我试图在并行处理python脚本中写入相同的共享数组。

当我在课外进行时，在正常的脚本中，一切正常。但是当我尝试通过一个类（使用相同的代码）来完成它时，我得到了 Runtime Error: SynchronizedArray objects should only be shared between processes through inheritance。

我的脚本如下（没有类）：

import numpy
import ctypes

from multiprocessing import Pool, Array, cpu_count

n = 2

total_costs_matrix_base = Array(ctypes.c_double, n*n)
total_costs_matrix = numpy.ctypeslib.as_array(
                     total_costs_matrix_base.get_obj())
total_costs_matrix = total_costs_matrix.reshape(n,n)


def set_total_costs_matrix( i, j, def_param = total_costs_matrix_base):
    total_costs_matrix[i,j] = i * j

if __name__ == "__main__":

    pool = Pool(processes=cpu_count())
    iterable = []

    for i in range(n):
        for j in range(i+1,n):
            iterable.append((i,j))
    pool.starmap(set_total_costs_matrix, iterable)
    total_costs_matrix.dump('some/path/to/file')

该脚本效果很好。没有的是以下（使用一个类）：

import numpy
import ctypes

from multiprocessing import Pool, Array, cpu_count

class CostComputation(object):
    """Computes the cost matrix."""

    def __init__(self):
        self.n = 2

        self.total_costs_matrix_base = Array(ctypes.c_double, self.n*self.n)
        self.total_costs_matrix = numpy.ctypeslib.as_array(
                             self.total_costs_matrix_base.get_obj())
        self.total_costs_matrix = self.total_costs_matrix.reshape(self.n,self.n)


    def set_total_costs_matrix(self, i, j, def_param = None):
        def_param = self.total_costs_matrix_base
        self.total_costs_matrix[i,j] = i * j


    def write_cost_matrix(self):
        pool = Pool(processes=cpu_count())
        iterable = []

        for i in range(self.n):
            for j in range(i+1,self.n):
                iterable.append((i,j))
        pool.starmap(self.set_total_costs_matrix, iterable)
        self.total_costs_matrix.dump('some/path/to/file')

在此之后，我会在创建write_cost_matrix的实例后从其他文件中调用CostComputation。

我看过this answer，但仍然无法解决我的问题。

我在Mac OSX Yosemite 10.10.4中使用Python 3.4.2。

修改
使用CostComputation类时，我使用的脚本是：

from cost_computation import CostComputation

cc = CostComputation()
cc.write_costs_matrix()

整个错误是：

Traceback (most recent call last):
  File "app.py", line 65, in <module>
    cc.write_cost_matrix()
  File "/path/to/cost_computation.py", line 75, in write_cost_matrix
    pool.starmap(self.set_total_costs_matrix, iterable)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 268, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
    put(task)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/sharedctypes.py", line 192, in __reduce__
    assert_spawning(self)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 347, in assert_spawning
    ' through inheritance' % type(obj).__name__
RuntimeError: SynchronizedArray objects should only be shared between processes through inheritance

Answer 1

尝试创建仅包含共享数据的第二个类。然后在主类中使用该类对象。