Question

我正在构建一个多线程应用程序。

我已经设置了一个threadPool。 [从队列中获取数据的大小为N和N的队列]

完成所有任务后，我使用

tasks.join()

其中任务是队列。

应用程序似乎运行顺利，直到某些时候突然出现（例如20分钟后），它会以错误终止

thread.error: can't start new thread

有什么想法吗？

编辑：线程是守护进程线程，代码如下：

while True:
    t0 = time.time()
    keyword_statuses = DBSession.query(KeywordStatus).filter(KeywordStatus.status==0).options(joinedload(KeywordStatus.keyword)).with_lockmode("update").limit(100)
    if keyword_statuses.count() == 0:
        DBSession.commit()
        break

    for kw_status in keyword_statuses:
       kw_status.status = 1
       DBSession.commit()

    t0 = time.time()
    w = SWorker(threads_no=32, network_server='http://192.168.1.242:8180/', keywords=keyword_statuses, cities=cities, saver=MySqlRawSave(DBSession), loglevel='debug')

    w.work()

print 'finished'

当守护程序线程被杀死时？当应用程序完成或work（）完成时？

查看线程池和工作者（来自配方）

from Queue import Queue
from threading import Thread, Event, current_thread
import time

event = Event()

class Worker(Thread):
    """Thread executing tasks from a given tasks queue"""

    def __init__(self, tasks):
        Thread.__init__(self)

        self.tasks = tasks
        self.daemon = True
        self.start()


    def run(self):
        '''Start processing tasks from the queue'''
        while True:
            event.wait()
            #time.sleep(0.1)
            try:
                func, args, callback = self.tasks.get()
            except Exception, e:
                print str(e)
                return
            else:
                if callback is None:
                    func(args)
                else:
                    callback(func(args))

                self.tasks.task_done()

class ThreadPool:
    """Pool of threads consuming tasks from a queue"""

    def __init__(self, num_threads):
        self.tasks = Queue(num_threads)
        for _ in range(num_threads): Worker(self.tasks)


    def add_task(self, func, args=None, callback=None):
        ''''Add a task to the queue'''
        self.tasks.put((func, args, callback))


    def wait_completion(self):
        '''Wait for completion of all the tasks in the queue'''
        self.tasks.join()


    def broadcast_block_event(self):
        '''blocks running threads'''
        event.clear()


    def broadcast_unblock_event(self):
        '''unblocks running threads'''
        event.set()


    def get_event(self):
        '''returns the event object'''
        return event

ALSo可能是问题，因为我在循环中创建了SWorker对象？旧SWorker会发生什么（垃圾收集？）？

Answer 1

仍然没有足够的代码来本地化问题，但我确信这是因为你没有利用线程并开始过多的线程。您是否从Queue python文档http://docs.python.org/library/queue.html（页面底部）中看到了规范示例？

我可以使用以下代码重现您的问题：

import threading
import Queue

q = Queue.Queue()

def worker():
    item = q.get(block=True)  # sleeps forever for now
    do_work(item)
    q.task_done()

# create infinite number of workers threads and fails
# after some time with "error: can't start new thread"
while True:
    t = threading.Thread(target=worker)
    t.start()
q.join() # newer reached this

相反，您必须创建具有已知线程数的线程轮询，并将数据放入队列中，如：

q = Queue()

def worker():
    while True:
        item = q.get()
        do_work(item)
        q.task_done()

for i in range(num_worker_threads):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done

UPD：如果你需要停止一些线程，你可以为它添加一个标志或发送一个特殊标记意味着“停止”为break while循环：

class Worker(Thread):
    break_msg = object() # just uniq mark sign

    def __init__(self):
        self.continue = True

    def run():
        while self.continue:  # can stop and destroy thread, (var 1)
            msg = queue.get(block=True)
            if msg == self.break_msg:
                return  # will stop and destroy thread (var 2)
            do_work()
            queue.task_done()

workers = [Worker() for _ in xrange(num_workers)]
for w in workers:
    w.start()
for task in tasks:
    queue.put(task)

for _ in xrange(num_workers):
    queue.put(Worker.break_msg) # stop thread after all tasks done. Need as many messages as many threads you have
OR
queue.join() # wait until all tasks done
for w in workers:
    w.continue = False
    w.put(None)

python无法启动新线程

1 个答案: