制作一个Python装饰器,将任何@classmethod函数转换为一个并行处理值向量的函数

时间:2015-12-11 21:11:09

标签: python python-3.x multiprocessing decorator vectorization

这是一个基本的Python问题。我试图用我自己的装饰器@classmethod来装饰my_decorator,它将任何函数转换为一个函数,该函数接受原始函数的参数列表作为参数。这是一个例子。

您有一个函数foo(),如果您想按顺序,参数abc运行它。你会做的

result_a = foo(a)
result_b = foo(b)
result_c = foo(c)

你可以创建一个适用于参数列表的函数L=[a, b, c],然后就可以了

results = foo_on_list([a, b, c])

我想为任何函数foo()制作一个装饰器。更重要的是,我希望使用multiprocessing库能够foo(a)异步执行foo(b)。我写了一个很好的装饰器,它适用于静态函数,但我无法概括为必须采用clsself参数的函数。这是一个例子。

import multiprocessing as mp

def my_decorator(method):
    def method_in_vector_form(obj, L, *args, **kwargs):
        pool = mp.Pool(2)
        print('Starting decorator')
        results = [pool.apply_async(method, args=(obj, value)) for value in L]
        pool.close()
        pool.join()
        return [r.get() for r in results]
    return method_in_vector_form

class Foo(object):        
    @classmethod
    def boo(cls, i):
        print('Running from Foo\'s boo' + i)

    @classmethod
    @my_decorator
    def bar(cls, chunk_id):
        print('Running from Foo\'s bar')
        cls.boo(chunk_id)
        return chunk_id * 2

class Child(Foo):
    @classmethod
    def boo(cls, i):
        print('Running from Child\'s version of boo' + i)

if __name__ == '__main__':
    print(Foo.bar([1, 2, 3]))
    print(Child.bar([1, 2, 3]))

我期待以下输出(由于异步性可能会出现故障)。

Starting decorator
Running from Foo\'s bar
Running from Foo\'s boo 1
Running from Foo\'s bar
Running from Foo\'s boo 2
Running from Foo\'s bar
Running from Foo\'s boo 3
[2, 4, 6]
Starting decorator
Running from Foo\'s bar
Running from Child\'s version of boo 1
Running from Foo\'s bar
Running from Child\'s version of boo 2
Running from Foo\'s bar
Running from Child\'s version of boo 3
[2, 4, 6]

这是当前的输出。

Starting decorator
Traceback (most recent call last):
  File "C:\....\dec_example.py", line 36, in <module>
    print(Foo.bar([1, 2, 3]))
  File "C:\....\TextFromPdf\dec_example.py", line 10, in method_in_vector_form
    return [r.get() for r in results]
  File "C:\....\dec_example.py", line 10, in <listcomp>
    return [r.get() for r in results]
  File "C:\....\pool.py", line 599, in get
    raise self._value
  File "C:\....\pool.py", line 383, in _handle_tasks
    put(task)
  File "C:\....\connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "C:\....\reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function Foo.bar at 0x0000000002897158>: attribute lookup bar on __main__ failed

0 个答案:

没有答案