我遇到了一个我无法解决的问题,它与多处理相关联,并在装饰器中使用它。
当我使用多处理调用方法run_in_parallels
时,我收到错误:
Can't pickle <function run_testcase at 0x00000000027789C8>: it's not found as __main__.run_testcase
调用发生在装饰器内部,然后是上述问题。在没有装饰器的情况下调用相同方法run_in_parallels
时,所有工作正常。
这个问题的原因是什么?
file: w_PythonHelper.py
desc:函数'run_in_parallel'用于同时运行多个进程。结束操作的第一种方法会阻止其他操作。
from multiprocessing import Process,Event
class ExtProcess(Process):
def __init__(self, event,*args,**kwargs):
self.event=event
Process.__init__(self,*args,**kwargs)
def run(self):
Process.run(self)
self.event.set()
class PythonHelper(object):
@staticmethod
def run_in_parallel(*functions):
event=Event()
processes=dict()
for function in functions:
fname=function[0]
try:fargs=function[1]
except:fargs=list()
try:fproc=function[2]
except:fproc=1
for i in range(fproc):
process=ExtProcess(event,target=fname,args=fargs)
process.start()
processes[process.pid]=process
event.wait()
for process in processes.values():
process.terminate()
for process in processes.values():
process.join()
档案:w_Recorder.py
desc:函数'capture'用于获取屏幕截图
from PIL import ImageGrab
import time
class Recorder(object):
def capture(self):
ImageGrab.grab().save("{f}.{e}".format(f=time.time(),e="png"))
档案:w_Decorators.py
desc:并行执行给定函数以及类'Recorder'的方法'capture'
from w_Recorder import Recorder
from w_PythonHelper import PythonHelper
def check(function):
def wrapper(*args):
try:
recorder=Recorder()
PythonHelper.run_in_parallel([function,args],[recorder.capture])
print("success")
except Exception as e:
print("failure: {}".format(e))
return function
return wrapper
档案: w_Logger.py
desc:主程序(生成错误)
from w_Decorators import check
import time
class Logger(object):
@check
def run_testcase(self):
# example function (runtime: 20s)
for i in range(20):
print("number: {}".format(i))
time.sleep(1)
def run_logger(self):
self.run_testcase()
if __name__=="__main__":
logger=Logger()
logger.run_logger()
档案: w_Logger.py
desc:主程序(核心工作)
from w_PythonHelper import PythonHelper
from w_Recorder import Recorder
import time
class Logger(object):
def run_testcase(self):
# example function (runtime: 20s)
for i in range(20):
print("number: {}".format(i))
time.sleep(1)
def run_logger(self):
recorder=Recorder()
PythonHelper.run_in_parallel([self.run_testcase],[recorder.capture])
if __name__=="__main__":
logger=Logger()
logger.run_logger()
这两种情况下相同的方法有何不同?
编辑: 有没有人有想法解决这个问题(这是Python的错误)?如果没有,也许有人知道在应用程序运行时捕获屏幕截图的好方法吗?
实际上我发现了类似的问题:multiprocessing.Process subclass works on Linux but not Windows
答案是:To fix this, you can remove the process member.
,但我怎么能这样做呢。
在process.start()
run_in_parallel(*functions)
时调试错误
EDIT2: 喜欢 ivan_pozdeev 写道:我可以使用包装器作为函数,但不能将它用作装饰器。我有很多由这个装饰器装饰的函数,最简单的方法是在装饰器里面使用多处理。但不幸的是我无法解决这个问题。也许有人已经解决了类似的问题。我会很感激任何提示。
'run_in_parallel'功能就像我想要的那样。两个或多个函数并行运行,第一个函数完成,强制终止第二个函数。当我调用包装器(函数,* args)时,函数工作正常,当我把这个机制放在装饰器中时,我得到'不能发现功能...它找不到'错误。细节可以在上面找到
我的追溯:
Traceback (most recent call last):
File "C:\Interpreters\Python32\lib\pickle.py", line 679, in save_global
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'run_testcase'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\EskyTests\w_Logger.py", line 19, in <module>
logger.run_logger()
File "C:\EskyTests\w_Logger.py", line 14, in run_logger
self.run_testcase()
File "C:\EskyTests\w_Decorators.py", line 14, in wrapper
PythonHelper.run_in_parallel([function,args],[recorder.capture])
File "C:\EskyTests\w_PythonHelper.py", line 25, in run_in_parallel
process.start()
File "C:\Interpreters\Python32\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 267, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 190, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Interpreters\Python32\lib\pickle.py", line 237, in dump
self.save(obj)
File "C:\Interpreters\Python32\lib\pickle.py", line 344, in save
self.save_reduce(obj=obj, *rv)
File "C:\Interpreters\Python32\lib\pickle.py", line 432, in save_reduce
save(state)
File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Interpreters\Python32\lib\pickle.py", line 623, in save_dict
self._batch_setitems(obj.items())
File "C:\Interpreters\Python32\lib\pickle.py", line 656, in _batch_setitems
save(v)
File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Interpreters\Python32\lib\pickle.py", line 683, in save_global
(obj, module, name))
_pickle.PicklingError: Can't pickle <function run_testcase at 0x00000000027725C8>: it's not found as __main__.run_testcase
答案 0 :(得分:4)
您传递给Process.__init__()
的功能在Windows中无法选取。阅读16.6 multiprocessing - Programming guidelines - Windows。
关于你使用顶级函数的错误 - 我怀疑你定义它的方式,它每次生成的方式都不同,因此在孩子中真的是“不同的对象”。 我建议传递一个简单的顶级函数,如果你确实需要这种复杂度,可以使用反射来调用更新:这没有帮助run_testcase
。
更新
我通过取消run_testcase
,run_in_parallel
和capture
取消了这项工作。 @check
装饰器已替换为def wrapper(function,*args)
并具有相同的功能:
import traceback
def wrapper(function,*args):
try:
recorder=Recorder()
PythonHelper().run_in_parallel([function,args],[recorder.capture])
print("success")
except Exception,e:
print("failure: "+traceback.format_exc(10))
主:
from w_Decorators import wrapper
if __name__=="__main__":
logger=Logger()
wrapper(logger.run_testcase)
就像我想的那样 - 装饰物品不可挑选。
答案 1 :(得分:1)
Tricky,但我认为正在发生的事情是check
在定义类时存储对未绑定方法的引用。当您调用self.run_testcase
时,工作示例使用对绑定方法run_logger
的引用。
我认为最好的办法是尝试使run_testcase
成为顶级函数,而不是类的方法。
此外,您的capture
函数可能无法达到预期效果 - 当定义函数时将存储当前时间,并且每个屏幕截图都将保存在前一个屏幕截图上。您可能想在函数内调用time.time()
。