Question

对于将实例方法传递给Pool的以下代码，脚本末尾的列表为空：

import time

from multiprocessing import Pool

class Hello:
    def __init__(self):
        self.result_list=[]

    def f(self,x,y):
        time.sleep(2)
        return x*y


    def log_result(self,result):
        # This is called whenever foo_pool(i) returns a result.
        # result_list is modified only by the main process, not the pool workers.
        print result
        self.result_list.append(result)

if __name__ == '__main__':
    pool = Pool()              # start 4 worker processes
    h=Hello()
    for i in range(10):
        pool.apply_async(h.f, args = (i,i, ), callback = h.log_result)
    pool.close()
    pool.join()
    print(h.result_list)

使用此代码，列表将按预期填充。

import multiprocessing as mp
import time

def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

def apply_async_with_callback():
    pool = mp.Pool()
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()

两者的不同之处是什么？为什么它不适用于实例方法？

Answer 1

如果你真的试图获取其中一个apply_async来电的结果，你会发现他们都失败了这个错误：

cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

这是因为在Python 2.x中，默认情况下实例方法不可选，因此尝试将实例方法h.f传递给工作进程失败。这实际上已在Python 3中修复，但您可以使用copy_reg模块轻松地将行为向后移植到Python 2：

import time

from multiprocessing import Pool
import copy_reg
import types

def _reduce_method(m):
    if m.__self__ is None:
        return getattr, (m.__class__, m.__func__.__name__)
    else:
        return getattr, (m.__self__, m.__func__.__name__)

copy_reg.pickle(types.MethodType, _reduce_method)

class Hello:
    def __init__(self):
        self.result_list=[]

    def f(self,x,y):
        time.sleep(2)
        return x*y


    def log_result(self, result):
        print(result)
        self.result_list.append(result)

if __name__ == '__main__':
    pool = Pool()
    h = Hello()
    for i in range(10):
        pool.apply_async(h.f, args = (i,i), callback=h.log_result)
    pool.close()
    pool.join()
    print(h.result_list)

输出：

0
4
49
25
1
16
36
9
64
81
[0, 4, 49, 25, 1, 16, 36, 9, 64, 81]

当使用实例方法和函数进行多处理时，为什么会得到不同的结果？

1 个答案: