Question

我遇到了多处理模块的麻烦。我已经使用了锁，但是下面的代码仍然不安全，我不知道为什么，计数器有时候不会等于100，我怎样才能修复代码让它安全？

import random
import threading
from multiprocessing import Pool, Manager

import time

lock = threading.Lock()


def a_complex_operation(counter):
    with lock:
        time.sleep(random.random())
        counter.value += 1


def main():
    pool = Pool(16)
    counter = Manager().Value('i', 0)

    for i in range(100):
        pool.apply_async(a_complex_operation, args=(counter,))

    pool.close()
    pool.join()
    if counter.value != 100:
        print "not equal 100, current value is: "+str(counter.value)


if __name__ == '__main__':
    count = 0
    while True:
        t1 = time.time()
        main()
        count += 1
        print "the " + str(count) + " loop, cost time: " + str(time.time() - t1)

输出将是：

the 1 loop, cost time: 4.1369998455
the 2 loop, cost time: 3.74100017548
the 3 loop, cost time: 3.92299985886
the 4 loop, cost time: 4.05500006676
not equal 100, current value is: 99
the 5 loop, cost time: 4.01900005341
the 6 loop, cost time: 4.14299988747

然后我测试Manager（）。list（）和Manager（）。值（＆＃39; i＆＃39;，0）

import random
from multiprocessing import Pool, Manager
import time


def a_complex_operation(list_, counter):
    for x in range(10):
        time.sleep(random.random()/10)
        list_.append(x)
        counter.value += 1


def main():
    pool = Pool(16)
    counter0 = 0
    list_ = Manager().list()
    counter = Manager().Value('i', 0)

    for i in range(100):
        pool.apply_async(a_complex_operation, args=(list_, counter))
        counter0 += 1

    pool.close()
    pool.join()
    if len(list_) != 1000:
        print "length of list is not equal 1000, current is:" + str(len(list_))
    if counter.value != 1000:
        print "value of counter is not equal 1000, current is :" + str(counter.value)


if __name__ == '__main__':
    counter = 0
    while True:
        counter += 1
        t1 = time.time()
        main()
        t2 = time.time()
        print "the " + str(counter) + " loop cost time: " + str(t2 - t1)

输出将是：

value of counter is not equal 1000, current is :916
the 1 loop cost time: 3.92299985886
value of counter is not equal 1000, current is :911
the 2 loop cost time: 3.98500013351
value of counter is not equal 1000, current is :925
the 3 loop cost time: 4.007999897
value of counter is not equal 1000, current is :913
the 4 loop cost time: 3.99399995804
value of counter is not equal 1000, current is :920
the 5 loop cost time: 4.09500002861
value of counter is not equal 1000, current is :915

我发现Manager（）。list（）是安全的，Manager（）。值（＆＃39; i＆＃39;，0）不安全，有趣，任何人都可以告诉我为什么Manager（）。list（）看起来喜欢安全吗？

Answer 1

您的子进程不继承锁对象。或者他们这样做，但他们是没有链接在一起的独立副本，不能用于任何事情。因此存在竞争条件并最终失败。

您可以通过Manager().Lock()解决此问题，因为您已经在使用Manager。

def a_complex_operation(counter, alock):
    with alock:
        time.sleep(random.random())
        counter.value += 1

def main():
    pool = Pool(16)
    ma = Manager()
    counter = ma.Value('i', 0)
    lock = ma.Lock()
    for i in range(100):
        pool.apply_async(a_complex_operation, args=(counter, lock))

这样可行（但是你的子程序现在会慢得多。每次运行大约需要50秒，平均每次运行需要50秒）。

但现在你的counter.value总是100。

Answer 2

我认为以下代码应该安全快速，感谢@Hannu和@gzc

import random
from multiprocessing import Pool, Manager
import time


def a_complex_operation(list_, counter, lock):
    for x in range(10):
        time.sleep(random.random() / 10)
        with lock:
            list_.append(x)
            counter.value += 1


def main():
    pool = Pool(16)
    list_ = Manager().list()
    counter = Manager().Value('i', 0)
    lock = Manager().Lock()

    for i in range(100):
        pool.apply_async(a_complex_operation, args=(list_, counter, lock))

    pool.close()
    pool.join()
    if len(list_) != 1000:
        print ">>> length of list is not equal 1000, current is:" + str(len(list_))
    elif len(list_) == 1000:
        print ">>> length of list is equal 1000"
    if counter.value != 1000:
        print "value of counter is not equal 1000, current is :" + str(counter.value)
    elif counter.value == 1000:
        print "value of counter is equal 1000"


if __name__ == '__main__':
    counter = 0
    while True:
        counter += 1
        t1 = time.time()
        main()
        t2 = time.time()
        print "the " + str(counter) + " loop cost time: " + str(t2 - t1)
        print "--------------------------------"

输出将是：

>>> length of list is equal 1000
value of counter is equal 1000
the 1 loop cost time: 3.78799986839
--------------------------------
>>> length of list is equal 1000
value of counter is equal 1000
the 2 loop cost time: 3.79299998283
--------------------------------
>>> length of list is equal 1000
value of counter is equal 1000
the 3 loop cost time: 3.78299999237
--------------------------------
>>> length of list is equal 1000
value of counter is equal 1000
the 4 loop cost time: 3.77500009537
--------------------------------

python多处理共享变量safe

2 个答案: