为什么multiprocessing.Process能够pickle装饰函数?

时间:2014-10-09 14:15:11

标签: python python-2.7 multiprocessing pickle python-decorators

所以我读了here,装饰函数无法被腌制。事实上:

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    pool = mp.Pool(processes=1)
    for _ in pool.imap_unordered(f, range(10)):
        pass
    pool.close()
    pool.join()
    print 'All done'

输出:

Traceback (most recent call last):
  File "deco0.py", line 19, in <module>
    for _ in pool.imap_unordered(f, range(10)):
  File "/Users/usualme/anaconda/lib/python2.7/multiprocessing/pool.py", line 659, in next
    raise value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

但现在我将map替换为Process

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    p = mp.Process(target=f, args=(1,))
    p.start()
    p.join()
    print 'All done'

输出:

1
Exception caught!
All done

为什么有效?不进行处理还需要腌制装饰函数吗?

1 个答案:

答案 0 :(得分:3)

它之所以有效,是因为您已经在Linux上运行,而Linux并不需要通过f来挑选Process.__init__在子进程中调用它。这是因为fos.fork通过fork继承。如果您在Windows上运行相同的代码(缺少f),或者尝试将Pool.apply传递给Pool/map / f(两者都需要腌制{{1}在子进程中调用它,你会得到一个错误。

无论您使用什么平台,此示例都将失败:

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    p = mp.Pool()
    p.apply(f, args=(1,))  # f needs to be pickled here.
    print 'All done'

输出:

1
Exception caught!
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed