Question

所以我读了here，装饰函数无法被腌制。事实上：

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    pool = mp.Pool(processes=1)
    for _ in pool.imap_unordered(f, range(10)):
        pass
    pool.close()
    pool.join()
    print 'All done'

输出：

Traceback (most recent call last):
  File "deco0.py", line 19, in <module>
    for _ in pool.imap_unordered(f, range(10)):
  File "/Users/usualme/anaconda/lib/python2.7/multiprocessing/pool.py", line 659, in next
    raise value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

但现在我将map替换为Process：

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    p = mp.Process(target=f, args=(1,))
    p.start()
    p.join()
    print 'All done'

输出：

1
Exception caught!
All done

为什么有效？不进行处理还需要腌制装饰函数吗？

Answer 1

它之所以有效，是因为您已经在Linux上运行，而Linux并不需要通过f来挑选Process.__init__在子进程中调用它。这是因为f由os.fork通过fork继承。如果您在Windows上运行相同的代码（缺少f），或者尝试将Pool.apply传递给Pool/map / f（两者都需要腌制{{1}在子进程中调用它，你会得到一个错误。

无论您使用什么平台，此示例都将失败：

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    p = mp.Pool()
    p.apply(f, args=(1,))  # f needs to be pickled here.
    print 'All done'

输出：

1
Exception caught!
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

为什么multiprocessing.Process能够pickle装饰函数？

1 个答案: