Question

我写了这个小应用程序来解决Python中的经典生产者/消费者问题。我知道我可以使用线程安全的队列机制来解决这个问题，但我有兴趣自己解决这个问题来学习。

from threading import Thread, Condition
from collections import deque
import random
import time

tasks = deque()
condition = Condition()

class Consumer(Thread):

    def process_task(self, task):
        self.log("Completed task " + str(task) )

    def get_task(self):
        # this is the only method that access the global object
        global tasks
        condition.acquire()
        if len(tasks) is 0:
            self.log("Nothing in the Queue... Sleeping!")
            condition.wait()
            return None
        task = tasks.popleft()
        condition.release();
        return task

    def sleep(self):
        self.log("Sleeping")
        time.sleep(1)

    def execute(self):
        while True:
            task = self.get_task()
            if task:
                self.process_task(task)

    def run(self):
        self.log("Started")
        self.execute()

    def log(self, msg):
        print("[ Consumer: {0} ] Consumed {1}".format(self.getName(), msg))

class Producer(Thread):

    def create_tasks(self):
        return [random.randint(0,100) for x in range(100)]

    def add_tasks(self, new_tasks):
        global tasks
        condition.acquire()
        tasks.extend(new_tasks)
        self.log("Produced " + str(len(new_tasks)) + "elements")
        self.log("Queue length : " + str(len(tasks)))
        self.notify_all()
        condition.release();

    def notify_all(self):
        condition.notify_all()
        self.log("Aweking consumers")

    def sleep(self):
        self.log("Sleeping")
        time.sleep(1)

    def execute(self):
        while True:
            new_tasks = self.create_tasks()
            self.add_tasks(new_tasks)
            self.sleep()

    def run(self):
        self.log("Started")
        self.execute()

    def log(self, msg):
        print("[ Producer: {0} ] {1}".format(self.getName(), msg))


Producer().start()
Producer().start()

Consumer().start()
Consumer().start()
Consumer().start()

该应用程序似乎正常工作。我的意思是我没有经历任何僵局或奇怪的例外。然而，我获得的结果并不是我所期望的，因为工作根本没有在不同的消费者中传播。我期待所有三个消费者线程都能并行完成一些工作，但这并没有发生。这是日志的摘录：

[ Consumer: Thread-4 ] Consumed Completed task 42
 ... 98 times the above line with a different task ( always thread 4)  ...
[ Consumer: Thread-4 ] Consumed Completed task 22
[ Consumer: Thread-4 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-5 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-3 ] Consumed Nothing in the Queue... Sleeping!
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 100
[ Producer: Thread-1 ] Aweking consumers
[ Producer: Thread-1 ] Sleeping
[ Consumer: Thread-3 ] Consumed Completed task 87
 ... 98 times the above line with a different task ( always thread 3)...
[ Consumer: Thread-3 ] Consumed Completed task 20
[ Consumer: Thread-3 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-4 ] Consumed Nothing in the Queue... Sleeping!
[ Consumer: Thread-5 ] Consumed Nothing in the Queue... Sleeping!

从上面的日志中可以看出，所有工作都由一个线程执行，而另外两个线程不执行任何操作。看起来他们从来没有机会跑。我尝试在执行任务后将消费者线程放入睡眠状态，但行为没有改变。

我的代码中是否有我看不到的错误？这种行为可能与着名的“GIL”有关吗？

感谢您的帮助。

Answer 1

你所看到的并不是一个真正的编码问题，它只是同一个线程通常赢得了重新获得条件互斥的竞争。线程有足够的时间来运行process_task并仍然赢得比赛，因为process_task几乎没有做任何事情。我不认为这实际上与GIL有关;我在Jython中运行了代码，它没有GIL，并且发生了相同的行为。

但绝对没有错误。当我在CPython中运行你的代码时，我偶尔会遇到多个线程：

[ Producer: Thread-1 ] Started
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 100
[ Producer: Thread-1 ] Aweking consumers
[ Producer: Thread-1 ] Sleeping
[ Producer: Thread-2 ] Started
[ Consumer: Thread-3 ] Consumed Started
[ Producer: Thread-2 ] Produced 100elements
[ Producer: Thread-2 ] Queue length : 200
[ Consumer: Thread-5 ] Consumed Started
[ Producer: Thread-2 ] Aweking consumers
[ Consumer: Thread-4 ] Consumed Started
[ Producer: Thread-2 ] Sleeping
[ Consumer: Thread-5 ] Consumed Completed task 53
[ Consumer: Thread-4 ] Consumed Completed task 73
[ Consumer: Thread-5 ] Consumed Completed task 83
[ Consumer: Thread-4 ] Consumed Completed task 71
[ Consumer: Thread-5 ] Consumed Completed task 67
[ Consumer: Thread-4 ] Consumed Completed task 7
[ Consumer: Thread-5 ] Consumed Completed task 34
[ Consumer: Thread-4 ] Consumed Completed task 68
[ Consumer: Thread-5 ] Consumed Completed task 15
[ Consumer: Thread-4 ] Consumed Completed task 29
[ Consumer: Thread-5 ] Consumed Completed task 20
... (4 and 5 continue to switch off)

另外，如果我在self.sleep()内调用process_task来模拟一些实际发生的实际工作，我会得到非常正常的结果，因此可以进行更加真实的测试：

[ Producer: Thread-1 ] Started
[ Producer: Thread-2 ] Started
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 100
[ Producer: Thread-1 ] Aweking consumers
[ Consumer: Thread-5 ] Consumed Started
[ Consumer: Thread-3 ] Consumed Started
[ Producer: Thread-1 ] Sleeping
[ Producer: Thread-2 ] Produced 100elements
[ Consumer: Thread-4 ] Consumed Started
[ Producer: Thread-2 ] Queue length : 200
[ Producer: Thread-2 ] Aweking consumers
[ Producer: Thread-2 ] Sleeping
[ Consumer: Thread-3 ] Consumed Sleeping
[ Consumer: Thread-5 ] Consumed Sleeping
[ Consumer: Thread-4 ] Consumed Sleeping
[ Producer: Thread-1 ] Produced 100elements
[ Consumer: Thread-3 ] Consumed Completed task 85
[ Consumer: Thread-5 ] Consumed Completed task 31
[ Producer: Thread-1 ] Queue length : 297
[ Consumer: Thread-4 ] Consumed Completed task 62
[ Producer: Thread-1 ] Aweking consumers
[ Producer: Thread-1 ] Sleeping
[ Producer: Thread-2 ] Produced 100elements
[ Producer: Thread-2 ] Queue length : 397
[ Producer: Thread-2 ] Aweking consumers
[ Producer: Thread-2 ] Sleeping
[ Consumer: Thread-3 ] Consumed Sleeping
[ Consumer: Thread-5 ] Consumed Sleeping
[ Consumer: Thread-4 ] Consumed Sleeping
[ Producer: Thread-1 ] Produced 100elements
[ Producer: Thread-1 ] Queue length : 494
[ Consumer: Thread-3 ] Consumed Completed task 99
[ Consumer: Thread-4 ] Consumed Completed task 58
[ Producer: Thread-1 ] Aweking consumers
[ Consumer: Thread-5 ] Consumed Completed task 18

具有许多线程的Python生产者/消费者

1 个答案: