我有这样的队列设置。我想遍历数据库查询中的所有项目并将它们传递到我的Downloader类。我的数据库连接一直在消失,最终我的程序就死了,因为我认为有太多的线程是开放的?
我收到错误:分段错误:11
有100K +项目。
如何正确解决此问题,一次处理几件事并加快处理速度?
class Downloader(threading.Thread):
"""Threaded File Downloader"""
def __init__(self, queue, db):
threading.Thread.__init__(self)
self.queue = queue
self.db = db
def remove_unicode(self, title):
try:
return unicodedata.normalize('NFKD', title).encode('ascii','ignore')
except:
return title
def run(self):
while True:
# gets the url from the queue
row = self.queue.get()
title = row[0]
etc...
def main(urls):
queue = Queue.Queue()
# create a thread pool and give them a queue
for i in range(5):
t = Downloader(queue, db)
t.setDaemon(True)
t.start()
# give the queue some data
i = 1
for url in urls:
print i
queue.put(url)
i+=1
# wait for the queue to finish
queue.join()
if __name__ == "__main__":
db = DatabaseUtil()
sql = 'SELECT `Title`, `Site` from `XYZ`'
titles = db.query(sql)
main(titles)