Question

我正在python中设计一个多线程例程，其中一个函数将一个函数添加到队列中，并且几个帮助线程并发将队列中的东西弹出，直到它为空。从理论上讲，这应该比单线程实现快得多，但在实际应用和我为这个问题设计的玩具示例中并非如此。我猜测原因是Queue对象的某些同步问题（根据python文档是线程安全的）但这只是猜测。任何有关一般优化的帮助都表示赞赏！

玩具示例的代码：

from Queue import Queue
import time
queue = Queue(maxsize=0)
counter = []

#called by an external main method, adds and removes from global queue in single thread
def single_thread():
    fillit()
    time_origin = "%.10f" % time.time()
    while not queue.empty():
        queue.get()
    time_end = "%.10f" % time.time()
    time_t = float(time_end) - float(time_origin)
    print "time of single threaded implementation: " + str(time_t) +"\n"

#called by an external main method; adds to queue and removes from queue in multiple threads
def multi_thread():
    fillit()
    time_origin = "%.10f" % time.time()
    spawn_threads(4)
    time_end = "%.10f" % time.time()
    time_t = float(time_end) - float(time_origin)
    print "time of multi threaded implementation: " + str(time_t) +"\n"

#Fills up the queue with 2^19 elements
def fillit():
    for i in range(2 ** 19):
        queue.put(i)

#Spawns n helper threads to help empty the queue
def spawn_threads(num_threads):
    for i in range(num_threads):
        counter.append(0)
        thread = myThread(i, "Thread-" + str(i))
        thread.setDaemon(True)
        thread.start()
    while not queue.empty():
        continue
    print "done with threads " + str(counter) +" elements removed!"

#THREADING SUPPORT CODE
import threading

class myThread (threading.Thread):
    def __init__(self, threadID, name):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
    def run(self): #each thread continues to empty the queue while it still has elements
        while not queue.empty():
            queue.get()
            global counter
            counter[self.threadID] +=1

以下结果：

time of single threaded implementation: 1.51300001144

done with threads [131077, 131070, 131071, 131070] elements removed!
time of multi threaded implementation: 7.77100014687

Answer 1

你在这里看到的是python GIL。 GIL确保每个进程只执行一个线程执行python代码，这使得多线程任务不受IO限制，而不是更快，请参阅this link。如果要进行真正的并发，请查看多处理而不是多线程。请注意，虽然多处理有更多的开销，但由于缺乏共享状态和旋转过程成本更高。

多线程队列操作不比单线程快

1 个答案: