Question

我有一个类实例列表，我想并行调用相同的实例方法，使用pathos能够pickle实例方法，真正的问题是当我想更改/添加属性到实例时，它不起作用，我认为这是因为子流程的酸洗是输入的深层复制。任何人都知道如何解决这个问题？我不想改变编写实例方法的方式（比如返回一个值并稍后将它放在一起）。

from joblib import Parallel, delayed
import pathos.multiprocessing as mp
# import multiprocessing as mp 
import random
import os

pool = mp.Pool(mp.cpu_count())

class Person(object):
    def __init__(self, name):
        self.name = name

    def print_name(self, num):
        self.num = num
        print "worker {}, person name {}, received int {}".format(os.getpid(), self.name, self.num)


people = [Person('a'),
          Person('b'),
          Person('c'),
          Person('d'),
          Person('e'),
          Person('f'),
          Person('g'),
          Person('h')]


for i, per in enumerate(people):
    pool.apply_async(Person.print_name, (per, i) )

pool.close()
pool.join()
print 'their number'
for per in people:
    print per.num

这是输出，找不到num属性，我认为这是因为对这些副本进行了更改。

In [1]: run delme.py
worker 13981, person name a, random int 0
worker 13982, person name b, random int 1
worker 13983, person name c, random int 2
worker 13984, person name d, random int 3
worker 13985, person name e, random int 4
worker 13986, person name f, random int 5
worker 13987, person name g, random int 6
worker 13988, person name h, random int 7
their number
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/chimerahomes/wenhoujx/brain_project/network_analysis/delme.py in <module>()
     39 print 'their number'
     40 for per in people:
---> 41     print per.num

AttributeError: 'Person' object has no attribute 'num'

在评论中建议我尝试从子进程返回self，但似乎是一个错误的错误，返回的self不是它的原始类型。请参阅以下代码：

import pickle
# from joblib import Parallel, delayed
import pathos.multiprocessing as mp
# import multiprocessing as mp 
import random
import os

pool = mp.Pool(mp.cpu_count())

class Person(object):
    def __init__(self, name):
        self.name = name

    def print_name(self, num):
        self.num = num
        print "worker {}, person name {}, received int {}".format(os.getpid(), self.name, self.num)
        # return itself and put everything together
        return self



people = [Person('a'),
          Person('b'),
          Person('c'),
          Person('d'),
          Person('e'),
          Person('f'),
          Person('g'),
          Person('h')]

# Parallel(n_jobs=-1)(delayed(Person.print_name)(per) for per in people)

res = []
for i, per in enumerate(people):
    res.append(pool.apply_async(Person.print_name, (per, i) ))

pool.close()
pool.join()
people = [rr.get() for rr in res]


print 'their number'
for per in people:
    print per.num

print isinstance(people[0], Person)

这是输出：

In [1]: run delme.py
worker 29963, person name a, received int 0
worker 29962, person name b, received int 1
worker 29964, person name c, received int 2
worker 29962, person name d, received int 3
worker 29966, person name e, received int 4
worker 29967, person name f, received int 5
worker 29966, person name g, received int 6
worker 29967, person name h, received int 7
their number
0
1
2
3
4
5
6
7
False

我使用默认的多处理包，它没有这样的问题。

Answer 1

问题是self.num是在子进程中分配的。多处理不会将原始对象传递回调用者。它确实传回了方法的返回码。因此，您可以直接或num传回self（但这通常效率低下，并且不会替换父级中的现有对象，只需创建一个新对象）。

Python多进程更改类实例到位

1 个答案: