Question

受this solution的启发，我试图在Python中建立工作进程的多处理池。这个想法是在工作程序实际开始工作之前将一些数据传递给工作程序，并最终重新使用它们。旨在最大程度减少每次调用工作进程所需的打包数据量（即，减少进程间通信开销）。我的MCVE看起来像这样：

import multiprocessing as mp
import numpy as np

def create_worker_context():
    global context # create "global" context in worker process
    context = {}

def init_worker_context(worker_id, some_const_array, DIMS, DTYPE):
    context.update({
        'worker_id': worker_id,
        'some_const_array': some_const_array,
        'tmp': np.zeros((DIMS, DIMS), dtype = DTYPE),
        }) # store context information in global namespace of worker
    return True # return True, verifying that the worker process received its data

class data_analysis:
    def __init__(self):
        self.DTYPE = 'float32'
        self.CPU_LEN = mp.cpu_count()
        self.DIMS = 100
        self.some_const_array = np.zeros((self.DIMS, self.DIMS), dtype = self.DTYPE)
        # Init multiprocessing pool
        self.cpu_pool = mp.Pool(processes = self.CPU_LEN, initializer = create_worker_context) # create pool and context in workers
        pool_results = [
            self.cpu_pool.apply_async(
                init_worker_context,
                args = (core_id, self.some_const_array, self.DIMS, self.DTYPE)
            ) for core_id in range(self.CPU_LEN)
            ] # pass information to workers' context
        result_batches = [result.get() for result in pool_results] # check if they got the information
        if not all(result_batches): # raise an error if things did not work
            raise SyntaxError('Workers could not be initialized ...')

    @staticmethod
    def process_batch(batch_data):
        context['tmp'][:,:] = context['some_const_array'] + batch_data # some fancy computation in worker
        return context['tmp'] # return result

    def process_all(self):
        input_data = np.arange(0, self.DIMS ** 2, dtype = self.DTYPE).reshape(self.DIMS, self.DIMS)
        pool_results = [
            self.cpu_pool.apply_async(
                data_analysis.process_batch,
                args = (input_data,)
            ) for _ in range(self.CPU_LEN)
            ] # let workers actually work
        result_batches = [result.get() for result in pool_results]
        for batch in result_batches[1:]:
            np.add(result_batches[0], batch, out = result_batches[0]) # reduce batches
        print(result_batches[0]) # show result

if __name__ == '__main__':
    data_analysis().process_all()

我正在CPython 3.6.6上运行以上代码。

奇怪的是……有时它有用，有时却没有。如果不起作用，则process_batch方法将引发异常，因为它无法在some_const_array中找到context作为键。完整的回溯看起来像这样：

(env) me@box:/path> python so.py 
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/path/so.py", line 37, in process_batch
    context['tmp'][:,:] = context['some_const_array'] + batch_data # some fancy computation in worker
KeyError: 'some_const_array'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/so.py", line 54, in <module>
    data_analysis().process_all()
  File "/path/so.py", line 48, in process_all
    result_batches = [result.get() for result in pool_results]
  File "/path/so.py", line 48, in <listcomp>
    result_batches = [result.get() for result in pool_results]
  File "/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
KeyError: 'some_const_array'

我很困惑。这里发生了什么？

如果我的context词典包含“较高类型”的对象，例如数据库驱动程序或类似驱动程序，我没有遇到这种问题。如果我的context字典包含基本的Python数据类型，集合或numpy数组，则只能重现此内容。

（是否有可能以更可靠的方式实现相同目标的更好方法？我知道我的方法被认为是hack ...）

Answer 1

您需要将init_worker_context的内容重新放置到initializer函数create_worker_context中。

您认为每一个工作者进程将运行init_worker_context的假设是造成您困惑的原因。您提交到池中的任务将被馈送到所有读取的工作进程的一个内部任务队列中。在您的情况下发生的事情是，某些工作进程完成了他们的任务，并再次竞争获得新任务。这样一来，一个工作进程可能会执行多个任务，而另一个工作进程却无法完成一个任务，这很可能发生。请记住，操作系统会为（工作进程的）线程安排运行时。

尝试访问多处理中的持久性数据时获取不稳定的运行时异常。

1 个答案: