我在一个类中有一个方法,该方法需要循环执行许多工作,我想将工作分散到我的所有核心上。
我编写了以下代码,如果我使用普通的map
,但使用pool.map
时会返回错误。
import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)
class OtherClass:
def run(sentence, graph):
return False
class SomeClass:
def __init__(self):
self.sentences = [["Some string"]]
self.graphs = ["string"]
def some_method(self):
other = OtherClass()
def single(params):
sentences, graph = params
return [other.run(sentence, graph) for sentence in sentences]
return list(pool.map(single, zip(self.sentences, self.graphs)))
SomeClass().some_method()
错误:
AttributeError:无法腌制本地对象'SomeClass.some_method..single'
为什么它不能腌制single
?我什至尝试将single
移到全局模块作用域(不在类内部-使其独立于上下文):
import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)
class OtherClass:
def run(sentence, graph):
return False
def single(params):
other = OtherClass()
sentences, graph = params
return [other.run(sentence, graph) for sentence in sentences]
class SomeClass:
def __init__(self):
self.sentences = [["Some string"]]
self.graphs = ["string"]
def some_method(self):
return list(pool.map(single, zip(self.sentences, self.graphs)))
SomeClass().some_method()
,我收到以下错误消息:
AttributeError:无法从'... / test.py'
获取模块' main '上的属性'single'
答案 0 :(得分:8)
我不小心发现了一个非常讨厌的解决方案。只要你
使用def
语句。如果声明该函数,则要在用于解决该问题的函数的开头使用Pool.map
关键字在global
中使用。但是我不会在严肃的应用程序中依靠它?
import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)
class OtherClass:
def run(sentence, graph):
return False
class SomeClass:
def __init__(self):
self.sentences = [["Some string"]]
self.graphs = ["string"]
def some_method(self):
global single # This is ugly, but does the trick XD
other = OtherClass()
def single(params):
sentences, graph = params
return [other.run(sentence, graph) for sentence in sentences]
return list(pool.map(single, zip(self.sentences, self.graphs)))
SomeClass().some_method()
答案 1 :(得分:5)
要在定义函数和类之前启动池,这样子进程就不能继承任何代码。将您的池启动到最底部并用if __name__ == '__main__':
import multiprocessing
class OtherClass:
def run(self, sentence, graph):
return False
def single(params):
other = OtherClass()
sentences, graph = params
return [other.run(sentence, graph) for sentence in sentences]
class SomeClass:
def __init__(self):
self.sentences = [["Some string"]]
self.graphs = ["string"]
def some_method(self):
return list(pool.map(single, zip(self.sentences, self.graphs)))
if __name__ == '__main__': # <- prevent RuntimeError for 'spawn'
# and 'forkserver' start_methods
with multiprocessing.Pool(multiprocessing.cpu_count() - 1) as pool:
print(SomeClass().some_method())