Python多处理池映射:AttributeError:无法腌制本地对象

时间:2018-09-10 20:35:27

标签: python multiprocessing python-multiprocessing

我在一个类中有一个方法,该方法需要循环执行许多工作,我想将工作分散到我的所有核心上。

我编写了以下代码,如果我使用普通的map,但使用pool.map时会返回错误。

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      other = OtherClass()

      def single(params):
          sentences, graph = params
          return [other.run(sentence, graph) for sentence in sentences]

      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

错误:

  

AttributeError:无法腌制本地对象'SomeClass.some_method..single'

为什么它不能腌制single?我什至尝试将single移到全局模块作用域(不在类内部-使其独立于上下文):

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

,我收到以下错误消息:

  

AttributeError:无法从'... / test.py'

获取模块' main '上的属性'single'

2 个答案:

答案 0 :(得分:8)

我不小心发现了一个非常讨厌的解决方案。只要你 使用def语句。如果声明该函数,则要在用于解决该问题的函数的开头使用Pool.map关键字在global中使用。但是我不会在严肃的应用程序中依靠它?

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)

class OtherClass:
  def run(sentence, graph):
    return False

class SomeClass:
  def __init__(self):
    self.sentences = [["Some string"]]
    self.graphs = ["string"]

  def some_method(self):
      global single  # This is ugly, but does the trick XD

      other = OtherClass()

      def single(params):
          sentences, graph = params
          return [other.run(sentence, graph) for sentence in sentences]

      return list(pool.map(single, zip(self.sentences, self.graphs)))


SomeClass().some_method()

答案 1 :(得分:5)

要在定义函数和类之前启动池,这样子进程就不能继承任何代码。将您的池启动到最底部并用if __name__ == '__main__':

保护它
import multiprocessing

class OtherClass:
  def run(self, sentence, graph):
    return False


def single(params):
    other = OtherClass()
    sentences, graph = params
    return [other.run(sentence, graph) for sentence in sentences]

class SomeClass:
   def __init__(self):
       self.sentences = [["Some string"]]
       self.graphs = ["string"]

   def some_method(self):
      return list(pool.map(single, zip(self.sentences, self.graphs)))

if __name__ == '__main__':  # <- prevent RuntimeError for 'spawn'
    # and 'forkserver' start_methods
    with multiprocessing.Pool(multiprocessing.cpu_count() - 1) as pool:
        print(SomeClass().some_method())