在装饰器内使用多处理会产生错误:无法解决功能...它找不到

时间:2011-11-14 18:58:26

标签: python multiprocessing decorator pickle

我遇到了一个我无法解决的问题,它与多处理相关联,并在装饰器中使用它。

当我使用多处理调用方法run_in_parallels时,我收到错误:

Can't pickle <function run_testcase at 0x00000000027789C8>: it's not found as __main__.run_testcase

调用发生在装饰器内部,然后是上述问题。在没有装饰器的情况下调用相同方法run_in_parallels时,所有工作正常。

这个问题的原因是什么?


file: w_PythonHelper.py

desc:函数'run_in_parallel'用于同时运行多个进程。结束操作的第一种方法会阻止其他操作。

from multiprocessing import Process,Event

class ExtProcess(Process):
    def __init__(self, event,*args,**kwargs):
        self.event=event
        Process.__init__(self,*args,**kwargs)

    def run(self):
        Process.run(self)
        self.event.set()

class PythonHelper(object):
    @staticmethod
    def run_in_parallel(*functions):
        event=Event()
        processes=dict()
        for function in functions:
            fname=function[0]
            try:fargs=function[1]
            except:fargs=list()
            try:fproc=function[2]
            except:fproc=1
            for i in range(fproc):
                process=ExtProcess(event,target=fname,args=fargs)
                process.start()
                processes[process.pid]=process
        event.wait()
        for process in processes.values():
            process.terminate()
        for process in processes.values():
            process.join()

档案:w_Recorder.py

desc:函数'capture'用于获取屏幕截图

from PIL import ImageGrab
import time

class Recorder(object):
    def capture(self):
        ImageGrab.grab().save("{f}.{e}".format(f=time.time(),e="png"))

档案:w_Decorators.py

desc:并行执行给定函数以及类'Recorder'的方法'capture'

from w_Recorder import Recorder
from w_PythonHelper import PythonHelper

def check(function):
    def wrapper(*args):
        try:
            recorder=Recorder()
            PythonHelper.run_in_parallel([function,args],[recorder.capture])
            print("success")
        except Exception as e:
            print("failure: {}".format(e))
        return function
    return wrapper

档案: w_Logger.py

desc:主程序(生成错误)

from w_Decorators import check
import time

class Logger(object):

    @check
    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)

    def run_logger(self):
        self.run_testcase()


if __name__=="__main__":
    logger=Logger()
    logger.run_logger()

档案: w_Logger.py

desc:主程序(核心工作)

from w_PythonHelper import PythonHelper
from w_Recorder import Recorder
import time

class Logger(object):

    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)

    def run_logger(self):
        recorder=Recorder()
        PythonHelper.run_in_parallel([self.run_testcase],[recorder.capture])

if __name__=="__main__":
    logger=Logger()
    logger.run_logger()

这两种情况下相同的方法有何不同?


编辑: 有没有人有想法解决这个问题(这是Python的错误)?如果没有,也许有人知道在应用程序运行时捕获屏幕截图的好方法吗?

实际上我发现了类似的问题:multiprocessing.Process subclass works on Linux but not Windows

答案是:To fix this, you can remove the process member.,但我怎么能这样做呢。

process.start()

中调用run_in_parallel(*functions)时调试错误

EDIT2: 喜欢 ivan_pozdeev 写道:我可以使用包装器作为函数,但不能将它用作装饰器。我有很多由这个装饰器装饰的函数,最简单的方法是在装饰器里面使用多处理。但不幸的是我无法解决这个问题。也许有人已经解决了类似的问题。我会很感激任何提示。

'run_in_parallel'功能就像我想要的那样。两个或多个函数并行运行,第一个函数完成,强制终止第二个函数。当我调用包装器(函数,* args)时,函数工作正常,当我把这个机制放在装饰器中时,我得到'不能发现功能...它找不到'错误。细节可以在上面找到

我的追溯:

Traceback (most recent call last):
  File "C:\Interpreters\Python32\lib\pickle.py", line 679, in save_global
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'run_testcase'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\EskyTests\w_Logger.py", line 19, in <module>
    logger.run_logger()
  File "C:\EskyTests\w_Logger.py", line 14, in run_logger
    self.run_testcase()
  File "C:\EskyTests\w_Decorators.py", line 14, in wrapper
    PythonHelper.run_in_parallel([function,args],[recorder.capture])
  File "C:\EskyTests\w_PythonHelper.py", line 25, in run_in_parallel
    process.start()
  File "C:\Interpreters\Python32\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 267, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 190, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Interpreters\Python32\lib\pickle.py", line 237, in dump
    self.save(obj)
  File "C:\Interpreters\Python32\lib\pickle.py", line 344, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Interpreters\Python32\lib\pickle.py", line 432, in save_reduce
    save(state)
  File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Interpreters\Python32\lib\pickle.py", line 623, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Interpreters\Python32\lib\pickle.py", line 656, in _batch_setitems
    save(v)
  File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Interpreters\Python32\lib\pickle.py", line 683, in save_global
    (obj, module, name))
_pickle.PicklingError: Can't pickle <function run_testcase at 0x00000000027725C8>: it's not found as __main__.run_testcase

2 个答案:

答案 0 :(得分:4)

您传递给Process.__init__()的功能在Windows中无法选取。阅读16.6 multiprocessing - Programming guidelines - Windows

关于你使用顶级函数的错误 - 我怀疑你定义它的方式,它每次生成的方式都不同,因此在孩子中真的是“不同的对象”。 我建议传递一个简单的顶级函数,如果你确实需要这种复杂度,可以使用反射来调用run_testcase更新:这没有帮助

更新

我通过取消run_testcaserun_in_parallelcapture取消了这项工作。 @check装饰器已替换为def wrapper(function,*args)并具有相同的功能:

import traceback
def wrapper(function,*args):
    try:
        recorder=Recorder()
        PythonHelper().run_in_parallel([function,args],[recorder.capture])
        print("success")
    except Exception,e:
        print("failure: "+traceback.format_exc(10))

主:

from w_Decorators import wrapper

if __name__=="__main__":
    logger=Logger()
    wrapper(logger.run_testcase)

就像我想的那样 - 装饰物品不可挑选。

答案 1 :(得分:1)

Tricky,但我认为正在发生的事情是check在定义类时存储对未绑定方法的引用。当您调用self.run_testcase时,工作示例使用对绑定方法run_logger的引用。

我认为最好的办法是尝试使run_testcase成为顶级函数,而不是类的方法。

此外,您的capture函数可能无法达到预期效果 - 当定义函数时将存储当前时间,并且每个屏幕截图都将保存在前一个屏幕截图上。您可能想在函数内调用time.time()